:W&BᾺØŸM n2m3adwrª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0gåfF‚" w0sxdebgrzy1 n2m3adwrŠÝ n2m3adwrescheuller-uc-san-diego spatop-sweep"Ÿ _wandb‚{}  batch_size‚2048  dropout‚0.15537113659220475  hidden_dim‚128 ' learning_rate‚0.0001787803981085858  num_embedding_layers‚10Bdpfx_blocks_n2m3adwrbox8or7c7j$dp-spatop-sweepagent-blocks-v2-fb276Š ëáöÒˆ¨‡Cš3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64v}; n2m3adwr*  _wandb‚ {"runtime":0}  _runtime‚0éÐb_ÒÙ -Linux-6.8.0-124-generic-x86_64-with-glibc2.36CPython 3.12.10 ëáöÒ ×ê*-of*>/data/spatop/options_files/sweep_fixed_dp/config_n2m3adwr.json*-l* /scratch/logs*-n*dpfx_blocks_n2m3adwr2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb$dp-spatop-sweepagent-blocks-v2-fb276r/opt/conda/bin/pythonº x57gvdn3c9fte6x5ltii8qj0ca5hh7fgæPò2  requirements.txt=lùBW n2m3adwrZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64žŠY n2m3adwrZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64,jr* n2m3adwrb trainer/global_step2‡W%Í\  n2m3adwrZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64¹œÝ3/  n2m3adwrb*"trainer/global_step2K  n2m3adwrj: ëáöÒð—öî*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] öùÇ ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€€ý/€àñ‡Ç$ª€Àû”ÌÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-ed01d6ad-b673-30ac-3501-763022903663Ê13.2º x57gvdn3c9fte6x5ltii8qj0ca5hh7fgD“?“  n2m3adwrj  ëáöÒø×Æ—î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ /AÈ€\ n2m3adwrjÊ ëáöÒè°¶˜·Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.816 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 ÷22°q n2m3adwrj` ëáöÒؽë˜Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ~•{ n2m3adwrjj ëáöÒøµø˜Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ï½qq n2m3adwrj` ëáö҈ベPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ¢HÄÈt n2m3adwrjc ëáöÒ¸¾Š™Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). :²œn n2m3adwrj] ëáöÒС‘™Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). âsžg. n2m3adwr*œ  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"»×¢¶ë n2m3adwrjÙ ëáöÒ€ãí›È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. þMŒN n2m3adwrj¼ ëáöÒØÔ„œ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Aoròë n2m3adwrjÙ îáöÒ€º€¼È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. @m)ÏP n2m3adwrj¾ îáöÒÀ­™¼­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ¾cC‚(:¹ úáöÒÂÎÞ( disk./.usagePercent‚76.1937672997546 disk.nvme0n1.out‚ 1.548828125 disk.rbd1.out‚0$ memory_percent‚95.75116634368896 cpu‚27.028316631043484 proc.cpu.threads‚17% disk./.usageGB‚1169.7648124694824 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.nvme1c1n1.out‚ 0.6669921875 disk.dm-1.out‚ 0.24609375 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 0.6669921875( proc.memory.availableMB‚ 696.12890625# proc.memory.rssMB‚ 2820.30078125* proc.memory.percent‚17.213749885559082 network.sent‚32659 disk.md0.in‚0 disk.md0.out‚0.296875 disk.dm-0.out‚ 1.3154296875 disk.nvme0c0n1.in‚0" disk.nvme0c0n1.out‚ 1.548828125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 network.recv‚38094ÎÂœ›‚(:‘ úáöÒů«ß gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚51.0 gpu.0.powerWatts‚266.4' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.79999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w¢1½L‚(: ‰âöÒ½Þ€Þ) disk./.usagePercent‚76.19377301465687 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0" proc.memory.rssMB‚ 2820.3046875 network.sent‚41212 disk.dm-1.out‚ 0.2890625# disk.nvme0n1.out‚11.77685546875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0$ memory_percent‚95.75216770172119* proc.memory.percent‚17.213773727416992 disk.md0.in‚0 disk.md0.out‚ 0.5703125 disk.dm-0.in‚0 disk.dm-1.in‚0% disk.nvme0c0n1.out‚11.77685546875( proc.memory.availableMB‚ 695.96484375 cpu‚26.496864010636145 network.recv‚41983% disk./.usageGB‚1169.7649002075195 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 3.79736328125 disk.dm-0.out‚ 14.0185546875$ disk.nvme1c1n1.out‚ 3.79736328125 proc.cpu.threads‚17.86^‚(:“ ‰âöÒššÚÞ gpu.0.gpu‚95.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚51.0 gpu.0.powerWatts‚278.579' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.85966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜| ØM‚(:à ˜âöÒÆÓøÝ network.recv‚45968% disk./.usageGB‚1169.7644424438477 disk.md0.in‚0 disk.dm-1.out‚ 0.49365234375 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 695.55078125" proc.memory.rssMB‚ 2820.3046875 network.sent‚48381$ disk.nvme1c1n1.out‚ 4.71044921875 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚0 disk.dm-0.out‚ 14.5693359375 cpu‚26.261390073480502 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0" disk.nvme1n1.out‚ 4.71044921875 disk.md0.out‚1.171875 disk.dm-0.in‚0$ disk.nvme0c0n1.out‚ 12.9033203125$ memory_percent‚95.75469493865967* proc.memory.percent‚17.213773727416992) disk./.usagePercent‚76.19374319777546 disk.dm-1.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 12.9033203125 disk.rbd0.out‚0!x3N’‚(:ˆ ˜âöÒ“¬¸Þ gpu.0.gpu‚57.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚277.866' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.622 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆCy3b/ lr-AdamW/pg1"trainer/global_step2:JPnýQß3 b/ lr-AdamW/pg2"trainer/global_step2:JPZoèí-!) _wandbruntime‚46  _runtime‚46ào1€À"» trainer/global_step‚49 % _timestamp‚1.7845250823422074e+09 _runtime‚ 46.610838202 _step‚0 & lr-AdamW/pg1‚2.190059876830176e-05 & lr-AdamW/pg2‚2.190059876830176e-05z Â#½ & lr-AdamW/pg1‚2.190059876830176e-05 & lr-AdamW/pg2‚2.190059876830176e-05 trainer/global_step‚49 % _timestamp‚1.7845250823422074e+09 _runtime‚ 46.610838202 _step‚0¼ÓÊg$bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP„ ­ l%bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPyU:·g&bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPñ{‘øj'bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP h_ìj(bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP;¤n)bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP334ti*be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP˜ÁŠÃi+be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP#4COn,bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPÛYß{n-bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP|ŸØ«b.b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPW³—Õn/bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPŒÒ„—k0bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPÛ¤Vq1bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPO¤¸êg2bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP\Ï\rf3bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPëÔØØc4b_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP4Oz)fba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPb`Åjf?bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP®.Ên@bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP]¯QkAbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP6¯£…iBbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP,4koCbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPkÕ§üdDb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP›Eµ xEbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP#¢UnFbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPYûO?iGbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP—Ì`hHbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP  yæaIb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP‹Å”Î_Jb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP›øšájKbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP ‰¤ tLbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPˆúžRiMbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP¾ûþ~lNbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP¤Ç .bOb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP'‚—pPbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP\GômQbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP:ÌÖhRbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPZ1—êsSbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP‰ƒrTbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP‰ò¸x^UbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP ˆÝCsVbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP§ GzfWbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPêôÝ=nXbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP”ÑÔrYbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP¾ ÏxZbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPŽ¤.ja[b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP*·_i\be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPfy{e]ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPªÍZ‘p^bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPmØ{n_bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP‡rg`bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPÜ.Ø\oabk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPveh(tbbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPíÅ9qccb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPQ€ìghbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP»¯Åähibd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP’sC]jbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPнÅmkbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPqÿÞmklbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP¹ å^mbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPÔÞÝpnbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPZ”ö-lobh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPô)SHopbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPn“fqbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP'jrÞirbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPõ·2lsbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPÉ[–0qtbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP9{ÎZqubm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP¶Q§Ôkvbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP¢ˆ½èfwbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPŒ^-Äpxbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP€ô jybf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPXÂ&gzbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPBÏm{bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPæŸrv|br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPbècœn}bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP'¶‹j~bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPï{ynbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPr™r€bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPÞ÷nbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPléžui‚bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP…_>>kƒbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP,Éïa„b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPhé~DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP7ñ1tšbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP¯&:t›bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÀWò½qœbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPáÚ„€gbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP eð€džb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7505609047552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7449317194240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7449300154880 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 57331799040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7505609047552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7463659692888 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 56222624966 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7505967331328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561205760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚551 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56650137088 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚491944 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99157504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515263 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚393555 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚491944 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1844693045248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚393553 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚393555 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 715023360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚478 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7449300154880 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 56308892672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 108823040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117769728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.c3Aýñ urrent‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7449317194240 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 56650137088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚49 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚482152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561205760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784525082346318e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534&‹/‹'¡…O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚393555 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 715023360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚478 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7449300154880 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 56308892672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 108823040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117769728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7449317194240 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 56650137088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚49 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚482152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561205760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784525082346318e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7407079468816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873433600 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 57430956544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚875705 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚482152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 1787243476480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 56308892672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1844575275520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7407096508176 UKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92316 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56563184712 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚393553 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚120147 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1787262088704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7505967331328 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7505609047552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7449317194240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7449300154880 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 57331799040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7505609047552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7463659692888 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 56222624966 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7505967331328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561205760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚551 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56650137088 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚491944 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99157504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515263 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚393555 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚491944 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1844693045248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚393553 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180a¹¦Le¢b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP}í>f£ba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP²mh¤bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPEºÚe¥b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP¦Ú@le¦b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPóHPªm§bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPžË¤g¨bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPQÿÖdf©ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP·f†bªb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPΨdDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP ³Qkg±bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP!Ü„i²bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPsdwh³bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JP7Ó'j´be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPDóÝ]dµb_ +g¶bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPîó‘p·bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPœ¨Ñ¯m¸bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPèôÉõm¹bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPðõåÅsºbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPâämoV»bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPXuÉ f¼ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPY‰«›h½bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPt¨…u¾bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP.Åï.o¿bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPzFtomÀbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPwMÙ³oÁbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPÎ>9oÂbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPac jÃbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPr¨õžuÄbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP¸×ܼmÅbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP‹b"pÆbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPŸ9ÆeÇb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPz‘šqÈbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPãÉflÉbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPZÒx.nÊbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPÕE8¾jËbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP8ÑznÌbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP¼ÃlÍbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPÑsë`Îb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPŠ{j fÏba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPìŒhrÐbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPñ”€\kÑbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPxçrÒbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPKq¢eÓb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPòT2hÔbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPÔÐhÕbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPýfÀbÖb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP †çÂd×b_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP?Èeåb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP8Ç{ næbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPvÀµwçbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPiF†ùkèbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPüKMÎhébc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPÀ‚L6mêbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPüÑAjëbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPÑNñdìb_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP=¼ómôbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPe8Dmõbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP×ô\öbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPJ­jig÷bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPG%Z9oøbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPcàÎ_ùbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPR”]^úbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPu¸Þbûb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPœ`b/mübh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP.÷!hýbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPº$™Wqþbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP>ë²iÿbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP°qÂßm€bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPÀt¢wbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP{’Û`‚b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPÛN[fƒba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPpÛè]„bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPäv&ýa…b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPÙÊýám†bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPLq‡bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPâ%ülˆbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPÂMâ†k‰bf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPþùá˜cŠb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPèœvÀh‹bc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPИdŒb_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845250829830086e+09 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚491808 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 57418991616 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 58543177216 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7655351939584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚501614 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚2 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚401348 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚122154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1820442358784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 SIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94050 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚478 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7612198748724 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚216201 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚893154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1878985536000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 58456609792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚501614 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7655351939584 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚216479 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86567424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚401348 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7597591696384 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚551 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚122429 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1878880356352 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 57331013218 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105179648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57760243200 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 108823040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚2 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚401348 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚122154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1820442358784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 SIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94050 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚478 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7612198748724 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚216201 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚893154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1878985536000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 58456609792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚501614 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7655351939584 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚216479 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86567424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚401348 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7597591696384 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚551 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚122429 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1878880356352 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 57331013218 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105179648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57760243200 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 108823040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845250829830086e+09 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚491808 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 57418991616 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 58543177216 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7655351939584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚501614 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528¢({[B b= loss/SRqqt1/detection_loss"trainer/global_step2:JPæùmÞA¡b< loss/FRt1/assignment_loss"trainer/global_step2:JP/*P†@¢b; loss/FRt2/detection_loss"trainer/global_step2:JP½®­KB£b= loss/SRqqt2/detection_loss"trainer/global_step2:JP)$çá@¤b; loss/FBt1/detection_loss"trainer/global_step2:JP9Íû@¥b; loss/FBt2/detection_loss"trainer/global_step2:JPÜÜää7¦b2 loss/total_loss"trainer/global_step2:JPà™ÏW-§b( epoch"trainer/global_step2:JPõ¤A¨b< loss/FRt2/assignment_loss"trainer/global_step2:JP§¬ÅtC©b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPÔwÝùCªb> loss/SRqqt2/assignment_loss"trainer/global_step2:JP6ƒ“A«b< loss/FBt1/assignment_loss"trainer/global_step2:JP âÈßA¬b< loss/FBt2/assignment_loss"trainer/global_step2:JP-¨L@­b; loss/FRt1/detection_loss"trainer/global_step2:JPùú6ï.®) _wandbruntime‚47  _runtime‚47ÍŠø$÷¯ñ .loss/FRt2/assignment_loss‚4.60714374987665 2loss/SRqqt1/assignment_loss‚4.2414488163067885 1loss/SRqqt2/assignment_loss‚4.162443161550151 0loss/FBt1/assignment_loss‚2.1394251693378794 0loss/FBt2/assignment_loss‚2.0884839946573432 /loss/FRt1/detection_loss‚2.0456066784968256 0loss/SRqqt1/detection_loss‚2.155117167651596 _runtime‚ 47.25349204 /loss/FRt1/assignment_loss‚5.352576925004382 /loss/FRt2/detection_loss‚0.7789414095965599 1loss/SRqqt2/detection_loss‚3.0455815228738934 /loss/FBt1/detection_loss‚24.973079975897615 /loss/FBt2/detection_loss‚14.358833271257504 %loss/total_loss‚69.94868184250718 trainer/global_step‚49 _step‚3 epoch‚0 # _timestamp‚1.78452508298482e+09ÇsZû°õ # _timestamp‚1.78452508298482e+09 .loss/FRt2/assignment_loss‚4.60714374987665 2loss/SRqqt1/assignment_loss‚4.2414488163067885 1loss/SRqqt2/assignment_loss‚4.162443161550151 0loss/FBt1/assignment_loss‚2.1394251693378794 0loss/FBt2/assignment_loss‚2.0884839946573432 /loss/FRt1/detection_loss‚2.0456066784968256 0loss/SRqqt1/detection_loss‚2.155117167651596 _runtime‚ 47.25349204 /loss/FRt1/assignment_loss‚5.352576925004382 /loss/FRt2/detection_loss‚0.7789414095965599 1loss/SRqqt2/detection_loss‚3.0455815228738934 /loss/FBt1/detection_loss‚24.973079975897615 /loss/FBt2/detection_loss‚14.358833271257504 %loss/total_loss‚69.94868184250718 trainer/global_step‚49 _step‚3 epoch‚0“Ä8»G±‚(:¼ §âöÒÝ°¹Þ$ disk.nvme1c1n1.out‚ 6.53857421875 disk.rbd1.in‚0 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚0# memory_percent‚95.7629919052124 proc.cpu.threads‚17 network.sent‚70652) disk./.usagePercent‚76.19374444014552 disk.dm-0.in‚0 disk.dm-1.out‚ 0.61865234375! disk.nvme0c0n1.out‚ 15.8671875( proc.memory.availableMB‚ 694.19140625* proc.memory.percent‚17.213988304138184 network.recv‚52011 disk.rbd1.out‚0" disk.nvme1n1.out‚ 6.53857421875 disk.md0.in‚0 disk.nvme0n1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2820.33984375 disk.nvme1c1n1.in‚0 disk.dm-0.out‚ 16.212890625 disk.dm-1.in‚0 disk.nvme0n1.out‚ 15.8671875 disk.rbd0.in‚0 cpu‚26.170397652136074 disk.md0.out‚ 2.6484375$ disk./.usageGB‚1169.764461517334‡Pž²‚(:“ §âöÒÖšƒß gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚282.403' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.13433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çr1H³‚(:½ ¶âöÒåÂêÝ$ memory_percent‚95.76513767242432 network.recv‚56326) disk./.usagePercent‚76.19374940962574 disk.md0.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0 cpu‚26.126373856287266 disk.md0.out‚3.140625 disk.nvme0c0n1.in‚0 disk.nvme0n1.out‚ 16.89453125 disk.rbd0.out‚0( proc.memory.availableMB‚ 693.83984375# proc.memory.rssMB‚ 2820.33984375* proc.memory.percent‚17.213988304138184 proc.cpu.threads‚17 disk.dm-0.out‚ 16.779296875 disk.dm-1.in‚0 disk.dm-1.out‚ 0.9208984375# disk.nvme1c1n1.out‚ 7.4345703125 disk.rbd1.out‚0 network.sent‚104846% disk./.usageGB‚1169.7645378112793 disk.dm-0.in‚0" disk.nvme0c0n1.out‚ 16.89453125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0! disk.nvme1n1.out‚ 7.4345703125€ž´‚(:“ ¶âöÒµ‹öÝ gpu.0.gpu‚40.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚284.936' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.97866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÄTHµ‚(:½ ÅâöÒžœ’Þ% disk./.usageGB‚1169.5280952453613 disk.rbd0.in‚0 disk.rbd1.out‚0 disk.nvme1n1.out‚ 8.498046875' proc.memory.availableMB‚ 694.0390625 proc.memory.rssMB‚ 2820.34375* proc.memory.percent‚17.214012145996094 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚0 cpu‚26.030828967536387 network.recv‚60119# disk.nvme0n1.out‚19.27490234375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚19.08837890625 network.sent‚110164) disk./.usagePercent‚76.17834849342852" disk.nvme1c1n1.out‚ 8.498046875 disk.md0.out‚3.65625 disk.dm-1.in‚0 disk.dm-1.out‚ 0.9599609375% disk.nvme0c0n1.out‚19.27490234375# memory_percent‚95.7639217376709BæþŸ¶‚(:” ÅâöÒÈâÝÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚123.107' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.035666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F[Ÿ.·) _wandbruntime‚92  _runtime‚92ÄXV Ã¸½ ' lr-AdamW/pg1‚4.4248148531874985e-05 ' lr-AdamW/pg2‚4.4248148531874985e-05 trainer/global_step‚99 % _timestamp‚1.7845251279746265e+09 _runtime‚ 92.242738581 _step‚4“žŒ»Ç¹Á ' lr-AdamW/pg1‚4.4248148531874985e-05 ' lr-AdamW/pg2‚4.4248148531874985e-05 trainer/global_step‚99 % _timestamp‚1.7845251279746265e+09 _runtime‚ 92.242738581 _step‚4^‰$#.º) _wandbruntime‚92  _runtime‚920§´ý£'»O HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14863042301440 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 113223076864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14863025262080 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112219238912 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚414742 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14778612842416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 111877994496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845251279756472e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚966804 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚577 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111740416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 117210624 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14890676279896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚783205 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚179013 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3446256980480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚966804 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14975261540352 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 111877994496 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚783205 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 111705838374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 113334817280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14778629881776 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚235729 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚783203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚976596 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14974903256576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130352640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 _runtime‚ 92.243854572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1750007 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚179016 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3559610409984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3446275592704 trainer/global_step‚99 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚976596 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14974903256576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14863025262080 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14975261540352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14863042301440 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 113223076864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚‰„"ÃO"14863025262080 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112219238912 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚414742 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14778612842416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 111877994496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845251279756472e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚966804 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚577 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111740416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 117210624 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14890676279896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚783205 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚179013 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3446256980480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚966804 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14975261540352 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 111877994496 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚783205 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 111705838374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 113334817280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14778629881776 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚235729 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚783203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚976596 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14974903256576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130352640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 _runtime‚ 92.243854572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1750007 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚179016 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3559610409984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3446275592704 trainer/global_step‚99 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚976596 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14974903256576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14863025262080 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14975261540352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 HDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3479455862784 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 114335435776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚976460 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 112988093440 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚180750 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚237986 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 114451363328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3593772686336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115927552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14926060549248 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15011299764224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚790996 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 113329345024 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚976460 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚577 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚418733 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845251286110811e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15124287857664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15011299764224 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113329345024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚180747 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 92.879411301 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15038857736514 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚790996 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚986266 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚238289 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3479437250560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚790998 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112814226626 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15011316803584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚986266 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184»Å#¦±&¿«M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845251286110811e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15124287857664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15011299764224 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113329345024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚180747 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 92.879411301 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15038857736514 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚790996 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚986266 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚238289 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3479437250560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚790998 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112814226626 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15011316803584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚986266 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3479455862784 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 114335435776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚976460 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 112988093440 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚180750 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚237986 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 114451363328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3593772686336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115927552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14926060549248 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15011299764224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚790996 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 113329345024 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚976460 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚577 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚418733 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858Êæø.À) _wandbruntime‚92  _runtime‚922?·áùÁó 1loss/SRqqt1/detection_loss‚1.3478508246955052 epoch‚0 _runtime‚ 92.880296263 1loss/SRqqt1/assignment_loss‚4.222268829442034 .loss/FBt1/detection_loss‚35.34113936949717 % _timestamp‚1.7845251286124718e+09 0loss/FBt1/assignment_loss‚1.3161300452131974 0loss/FBt2/assignment_loss‚1.0514645053415883 0loss/FRt2/detection_loss‚0.45388734314104384 0loss/SRqqt2/detection_loss‚4.322444106219336 %loss/total_loss‚71.61155577206617 _step‚7 /loss/FRt1/detection_loss‚1.5126160255427743 .loss/FBt2/detection_loss‚7.997958730706456 trainer/global_step‚99 /loss/FRt1/assignment_loss‚5.361590573833164 /loss/FRt2/assignment_loss‚4.572014410402522 1loss/SRqqt2/assignment_loss‚4.112191008031369 S,ýÂ÷ 1loss/SRqqt1/assignment_loss‚4.222268829442034 .loss/FBt1/detection_loss‚35.34113936949717 % _timestamp‚1.7845251286124718e+09 0loss/FBt1/assignment_loss‚1.3161300452131974 0loss/FBt2/assignment_loss‚1.0514645053415883 0loss/FRt2/detection_loss‚0.45388734314104384 0loss/SRqqt2/detection_loss‚4.322444106219336 %loss/total_loss‚71.61155577206617 _step‚7 /loss/FRt1/detection_loss‚1.5126160255427743 .loss/FBt2/detection_loss‚7.997958730706456 trainer/global_step‚99 /loss/FRt1/assignment_loss‚5.361590573833164 /loss/FRt2/assignment_loss‚4.572014410402522 1loss/SRqqt2/assignment_loss‚4.112191008031369 1loss/SRqqt1/detection_loss‚1.3478508246955052 epoch‚0 _runtime‚ 92.880296263$ÒŸZQÂ(:Æ ÔâöÒÙ æÝ% proc.memory.availableMB‚ 693.65625# proc.memory.rssMB‚ 2820.35546875 network.sent‚115460% disk./.usageGB‚1169.5277404785156 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0# memory_percent‚95.7662582397461 cpu‚25.955076003212465 disk.rbd1.in‚0" disk.nvme1n1.out‚ 9.28662109375 disk.md0.out‚ 4.19140625 disk.dm-0.in‚0 disk.dm-1.out‚ 1.07763671875 network.recv‚63915 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme0c0n1.out‚20.36376953125# disk.nvme0n1.out‚20.36376953125* proc.memory.percent‚17.214083671569824 proc.cpu.threads‚17) disk./.usagePercent‚76.17832538534543$ disk.nvme1c1n1.out‚ 9.28662109375 disk.dm-0.out‚19.70166015625 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0§ïªœÄ‚(:‘ ÔâöÒ€ýÛÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚280.303' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.43433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÛsÕQÅ‚(:Æ ãâöÒÂ’Þ network.recv‚68673 disk.rbd0.in‚0" disk.nvme1n1.out‚ 9.92333984375 disk.dm-1.out‚ 1.12451171875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0# proc.memory.rssMB‚ 2820.35546875) disk./.usagePercent‚76.16285638726897 disk.nvme1n1.in‚0 disk.md0.out‚ 4.50390625 disk.dm-0.out‚25.40771484375 disk.dm-1.in‚0% disk.nvme0c0n1.out‚26.15185546875* proc.memory.percent‚17.214083671569824 cpu‚25.8957125714227 network.sent‚163532 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0" memory_percent‚95.755934715271( proc.memory.availableMB‚ 695.34765625 proc.cpu.threads‚17% disk./.usageGB‚1169.2902526855469 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 9.92333984375 disk.md0.in‚0 disk.dm-0.in‚0# disk.nvme0n1.out‚26.15185546875¿ràèvÆ‚(:ˆ ãâöÒ•îçÞ gpu.0.gpu‚47.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚273.642' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.K<+›0.uncorrectedMemoryErrors‚0Q’ÁTÇ‚(:É òâöÒÑæ–Þ disk.dm-0.out‚26.01708984375) disk./.usagePercent‚76.16285837506106$ disk./.usageGB‚1169.290283203125 disk.rbd1.in‚0# disk.nvme1n1.out‚10.76806640625 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚10.76806640625 disk.rbd1.out‚0 disk.md0.in‚0 disk.md0.out‚ 5.08203125 disk.nvme0n1.in‚0# disk.nvme0n1.out‚27.31201171875 proc.cpu.threads‚17 network.sent‚168860 network.recv‚72586 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 1.27783203125% disk.nvme0c0n1.out‚27.31201171875$ memory_percent‚95.75619697570801* proc.memory.percent‚17.214083671569824 disk.dm-1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 695.3046875# proc.memory.rssMB‚ 2820.35546875 cpu‚25.84987920537753œ/áÿœÈ‚(:‘ òâöÒLJäÞ gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚281.758' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.91933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çgw\0É+ _wandbruntime‚136  _runtime‚136fzܘÂʼ & lr-AdamW/pg2‚6.659569829544821e-05 trainer/global_step‚149 $ _timestamp‚1.784525172407132e+09 _runtime‚ 136.675218025 _step‚8 & lr-AdamW/pg1‚6.659569829544821e-05 .´üÆËÀ & lr-AdamW/pg1‚6.659569829544821e-05 & lr-AdamW/pg2‚6.659569829544821e-05 trainer/global_step‚149 $ _timestamp‚1.784525172407132e+09 _runtime‚ 136.675218025 _step‚8WG›K0Ì+ _wandbruntime‚136  _runtime‚136—FÐC­'ͧO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1451456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22276750369280 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 167788340736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22317692866904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22276750369280 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1172853 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 167529611528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167447096320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5105270484480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚351757 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚617181 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22444555749376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5274467875328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1451456 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2624309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22150163255376 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167189051782 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22276767408640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 167447096320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22276767408640 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1461248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 117210624 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1461248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22444555749376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22444197465600 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚265713 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103351808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1172855 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚617473 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121964032 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 167788340736 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634103 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22150146216016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 _step‚9 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22444197465600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚577 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0.Þ’2±'ΫO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22276767408640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 167447096320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22276767408640 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1461248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 117210624 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1461248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22444555749376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22444197465600 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚265713 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103351808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1172855 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚617473 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121964032 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 167788340736 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634103 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22150146216016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 _step‚9 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22444197465600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚577 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1451456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22276750369280 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 167788340736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22317692866904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22276750369280 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1172853 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 167529611528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167447096320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5105270484480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚351757 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚617181 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22444555749376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5274467875328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1451456 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2624309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22150163255376 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167189051782 6êõP0Ï+ _wandbruntime‚137  _runtime‚1372ïR¸&вM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚621197 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚267450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1180646 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641758 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22297576883488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1461112 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1180646 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚621490 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126151168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1470918 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107538944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845251730325036e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2651566 _step‚10 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 117210624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 168297440034 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 170296782848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 168637999892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 137.300880224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚267447 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 170189243904 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452Ï-ÆqѶM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1180646 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641758 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICD÷æK$eviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22297576883488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1461112 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1180646 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚621490 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126151168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1470918 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107538944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845251730325036e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2651566 _step‚10 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 117210624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 168297440034 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 170296782848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 168637999892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 137.300880224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚267447 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 170189243904 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚621197 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚267450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 94›0Ò+ _wandbruntime‚137  _runtime‚137t´»çþÓø trainer/global_step‚149 _step‚11 2loss/SRqqt1/assignment_loss‚2.7561870628207394 1loss/FBt2/assignment_loss‚0.23563944597822406 /loss/FBt1/detection_loss‚30.802622952823377 _runtime‚ 137.301499265 /loss/FRt2/assignment_loss‚4.544554880926017 /loss/SRqqt2/detection_loss‚3.71013894404731 .loss/FBt2/detection_loss‚5.972551242669409 %loss/total_loss‚59.34860589688566 % _timestamp‚1.7845251730336616e+09 /loss/FRt1/assignment_loss‚5.312578439389902 0loss/FBt1/assignment_loss‚0.3536537780404784 0loss/FRt2/detection_loss‚0.38702250541333794 0loss/SRqqt1/detection_loss‚1.129334878508992 epoch‚0 2loss/SRqqt2/assignment_loss‚2.6072283557247258 /loss/FRt1/detection_loss‚1.5370934105431597bB£Ôü 2loss/SRqqt2/assignment_loss‚2.6072283557247258 /loss/FRt1/detection_loss‚1.5370934105431597 trainer/global_step‚149 _step‚11 2loss/SRqqt1/assignment_loss‚2.7561870628207394 1loss/FBt2/assignment_loss‚0.23563944597822406 /loss/FBt1/detection_loss‚30.802622952823377 _runtime‚ 137.301499265 /loss/FRt2/assignment_loss‚4.544554880926017 /loss/SRqqt2/detection_loss‚3.71013894404731 .loss/FBt2/detection_loss‚5.972551242669409 %loss/total_loss‚59.34860589688566 % _timestamp‚1.7845251730336616e+09 /loss/FRt1/assignment_loss‚5.312578439389902 0loss/FBt1/assignment_loss‚0.3536537780404784 0loss/FRt2/detection_loss‚0.38702250541333794 0loss/SRqqt1/detection_loss‚1.129334878508992 epoch‚0 ݲÞWQÕ‚(:Æ ãöÒÈÒöÝ" disk.nvme1n1.out‚ 14.2255859375( proc.memory.availableMB‚ 695.16015625 network.recv‚76606 disk.md0.in‚0# disk.nvme0n1.out‚41.10107421875 disk.rbd1.in‚0 disk.nvme1n1.in‚0* proc.memory.percent‚17.214083671569824 disk.md0.out‚5.390625 disk.dm-1.in‚0 disk.dm-1.out‚ 1.37939453125 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 14.2255859375$ memory_percent‚95.75707912445068 cpu‚25.81240388150473 network.sent‚173145 disk.dm-0.in‚0 disk.dm-0.out‚ 42.4892578125% disk.nvme0c0n1.out‚41.10107421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2820.35546875 proc.cpu.threads‚17) disk./.usagePercent‚76.16285638726897% disk./.usageGB‚1169.2902526855469 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0£ßñížÖ‚(:“ ãöÒ߶êÞ gpu.0.gpu‚44.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚188.425' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.80833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'‡²Pׂ(:Å ãöÒÕúùÝ disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚14.65478515625 cpu‚25.781045047898548 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 1.44189453125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14.65478515625 disk.md0.in‚0 network.sent‚220297 disk.md0.out‚ 5.65234375$ memory_percent‚95.75824737548828% proc.memory.availableMB‚ 694.96875 proc.cpu.threads‚17 disk.dm-0.out‚ 43.0322265625 network.recv‚81189( disk./.usagePercent‚76.1628633445413 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 41.8544921875# proc.memory.rssMB‚ 2820.35546875* proc.memory.percent‚17.214083671569824% disk./.usageGB‚1169.2903594970703 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 41.8544921875 disk.rbd0.in‚0 disk.rbd0.out‚0Û[®SœØ‚(:‘ ãöÒôÉÅÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚150.458' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚50.15266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—þ[Ù‚(:Ð ŸãöÒؘêÝ* proc.memory.percent‚17.214083671569824 proc.cpu.threads‚17% disk./.usageGB‚1169.7679481506348 disk.rbd1.in‚0 disk.nvme1n1.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚263.14599609375 disk.rbd0.out‚0 cpu‚25.754517210005428$ disk.nvme1n1.out‚220.02685546875 disk.md0.in‚0 disk.md0.out‚ 6.0703125! disk.dm-0.out‚468.62744140625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚263.14599609375 disk.rbd0.in‚0 network.sent‚226846) disk./.usagePercent‚76.19397154539227 disk.rbd1.out‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 1.60986328125$ memory_percent‚95.75955867767334( proc.memory.availableMB‚ 694.75390625# proc.memory.rssMB‚ 2820.35546875 network.recv‚85425 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚220.02685546875­óÎzžÚ‚(:“ ŸãöÒÄ‘ÁÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚154.591' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚51.53033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c„|ª0Û+ _wandbruntime‚181  _runtime‚181ëš8ËÄܾ & lr-AdamW/pg2‚8.894324805902144e-05 trainer/global_step‚199 % _timestamp‚1.7845252167574391e+09 _runtime‚ 181.025532849 _step‚12 & lr-AdamW/pg1‚8.894324805902144e-05G½€ÈÝ & lr-AdamW/pg2‚8.894324805902144e-05 trainer/global_step‚199 % _timestamp‚1.7845252167574391e+09 _runtime‚ 181.025532849 _step‚12 & lr-AdamW/pg1‚8.894324805902144e-05 Ó¤½0Þ+ _wandbruntime‚181  _runtime‚181vµ  ®'ߨO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1562505 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29690492515840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6989201732608 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 223357442560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29744709453912 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29913849958400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29913491674624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1562503 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6989325793792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1562505 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚467156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29521696628976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚819572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚819292 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚466879 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845252167585845e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1936108 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29744351854806 _step‚13 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1562503 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1936108 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1945900 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 224917744128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚352416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29521679589616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29690475476480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6764302600704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 223012824936 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29913849958400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124061184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223016198144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1945900 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6764283988480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29690492515840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29690475476480 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223016198144 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 222672265190 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508405 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 225023193088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚ 181.026799451 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29913491674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105448960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184ûÚ’§à¬O HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29744709453912 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29913849958400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29913491674624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1562503 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6989325793792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1562505 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚467156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29521696628976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚819572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚819292 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚466879 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845252167585845e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1936108 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29744351854806 _step‚13 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1562503 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1936108 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1945900 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 224917744128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚352416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29521679589616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29690475476480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6764302600704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 223012824936 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29913849958400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124061184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223016198144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1945900 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6764283988480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 JADeviceStatöôâñ sMonitor.on_train_batch_start/active.small_pool.current‚9792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29690492515840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29690475476480 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223016198144 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 222672265190 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508405 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 225023193088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚ 181.026799451 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29913491674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105448960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1562505 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29690492515840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6989201732608 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 223357442560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 <>Êæ0á+ _wandbruntime‚181  _runtime‚181e¶·&â±M SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚823269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29669127296448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29892890910530 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1570296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29669110257088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1955570 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚469122 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30063234566656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚14 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 224126297088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 224467548672 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3516060 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6797464258560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29893248509748 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29838767017984 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚823571 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7023620619264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130345472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1570298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30062876275712 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1570298 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚354147 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 223780653442 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525868 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 224467548672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1945764 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29838767017984 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30063234566656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344Li,»&ãµM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 $ _timestamp‚1.784525217390362e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29838767017984 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30063234566656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚823269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29669127296448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29892890910530 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1570296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29669110257088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1955570 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚469122 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30063234566656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚14 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 224126297088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 224467548672 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3516060 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6797464258560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29893248509748 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29838767017984 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚823571 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7023620619264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130345472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1570298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30062876275712 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1570298 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚354147 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 223780653442 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525868 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 224467548672 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1945764 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚439116 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8423297492480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37382785883648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37104200583680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37104200583680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1952155 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚17 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37104217623040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126158336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 278496038344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4372913 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4382707 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37104217623040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1952153 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272ëy´'ò®O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37104217623040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126158336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 278496038344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4372913 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4382707 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37104217623040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1952153 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37171726040920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2430552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 trainer/global_step‚249 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1952155 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280742208000 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 278155478598 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37382785883648 % _timestamp‚1.7845252620905428e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278926544384 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36893230002576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚439116 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8423297492480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37382785883648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37104200583680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37104200583680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1952155 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚17 HDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 280036650496 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8738336339456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37252475085824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1959946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2430416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37252492125184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 226.988859312 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2440222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103344640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 280036650496 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37532528775680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37532170484736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 279695398912 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37532528775680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1025435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8456477762560n•=tõ¸M MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2430416 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚440850 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8456496374784 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1959948 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121956864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚18 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37040660670048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 279263866850 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1959946 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37532528775680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1025435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8456477762560 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37320265096756 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1959948 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 280036650496 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8738336339456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37252475085824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1959946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2430416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.c§yÊJ urrent‚17039360 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37252492125184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 226.988859312 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2440222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103344640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 280036650496 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37532528775680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37532170484736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 279695398912 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2341803 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 334495646208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _runtime‚ 271.643605731 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44517925690880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44852080092672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44264763376176 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2341805 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2341803 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 334495646208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _runtime‚ 271.643605731 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44517925690880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44852080092672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44264763376176 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2341805 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 335605752320 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2349596 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 335264500736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 337856891392 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚527547 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1227662 UHDeviceStatsMonitor.y¡Å on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚527550 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10115491266560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2915068 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1227963 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5264664 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 337743060992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 335264500736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113830400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45001822984704 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚700115 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44746924084546 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 335087640116±3ÞÁ&‡»M F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5274472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2915068 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44666217232384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚700115 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44746924084546 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 335087640116 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2924874 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44747281683764 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44412177004288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 335605752320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44666200193024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2349598 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.784525308007573e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10453366770176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10453234327552 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 334747080258 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 335605752320 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2349596 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 335264500736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 337856891392 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚527547 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1227662 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚527550 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10115491266560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2915068 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1227963 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5264664 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 337743060992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 335264500736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113830400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45001822984704 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51931667837440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130352640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52321374301696 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚612513 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 389462465160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2731455 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3390064 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2731453 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 389723503616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51636279710416 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3399856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51931650798080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3390064 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚813582 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845253526175292e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1426098 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 389723503616 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚349 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52321374301696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52321732585472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51636296749776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51931650798080 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1425811 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12133910036480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52321732585472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51636296749776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51931650798080 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1425811 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12133910036480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51931667837440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130352640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52321374301696 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚612513 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 389462465160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2731455 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3390064 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2731453 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 389723503616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51636279710416 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3399856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51931650798080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3390064 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚813582 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845253526175292e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1426098 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 389723503616 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚349Sêh0—+ _wandbruntime‚317  _runtime‚317F\>Ú¾&˜¸M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 _step‚26 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 391174854144 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52471117193728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126151168 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12168078144512 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52174298270772 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52173940671554 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6148774 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚815877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 393573373952 trainer/global_step‚349 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 317.517919222 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2739248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11774523382784 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 390570853524 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52470758902784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52079925300224 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2739248 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3399720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51783710377888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11774504770560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52079942339584 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 390833602560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7845253532495723e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3409526 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2739246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1429838 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚815591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180ú}*žÂ&™¼M TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 390833602560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7845253532495723e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3409526 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2739246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1429838 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚815591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 _step‚26 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 391174854144 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52471117193728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126151168 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12168078144512 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52174298270772 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52173940671554 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6148774 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚815877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 393573373952 trainer/global_step‚349 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 317.517919222 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2739248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11774523382784 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 390570853524 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52470758902784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52079925300224 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2739248 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3399720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51783710377888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11774504770560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52079942339584 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584IÛ‚0š+ _wandbruntime‚317  _runtime‚317)õ Ãý›÷ 1loss/FBt2/assignment_loss‚0.44717330114771714 %loss/total_loss‚31.62658394373009 _runtime‚ 317.518499883 _step‚27 /loss/FRt1/assignment_loss‚4.578154294591554 /loss/FRt2/assignment_loss‚4.502057552996261 0loss/FBt1/assignment_loss‚0.2066004705254091 0loss/SRqqt2/detection_loss‚3.654343851468879 .loss/FBt1/detection_loss‚5.930492480348634 epoch‚0 trainer/global_step‚349 2loss/SRqqt1/assignment_loss‚1.6133241682733277 1loss/SRqqt2/assignment_loss‚1.850699184659105 /loss/FRt1/detection_loss‚1.4969823450558413 /loss/FRt2/detection_loss‚0.2726707145741066 1loss/SRqqt1/detection_loss‚1.0596074912957332 .loss/FBt2/detection_loss‚6.014478088793524 % _timestamp‚1.7845253532506754e+09«]¸Üœû .loss/FBt1/detection_loss‚5.930492480348634 epoch‚0 trainer/global_step‚349 2loss/SRqqt1/assignment_loss‚1.6133241682733277 1loss/SRqqt2/assignment_loss‚1.850699184659105 /loss/FRt1/detection_loss‚1.4969823450558413 /loss/FRt2/detection_loss‚0.2726707145741066 1loss/SRqqt1/detection_loss‚1.0596074912957332 .loss/FBt2/detection_loss‚6.014478088793524 % _timestamp‚1.7845253532506754e+09 1loss/FBt2/assignment_loss‚0.44717330114771714 %loss/total_loss‚31.62658394373009 _runtime‚ 317.518499883 _step‚27 /loss/FRt1/assignment_loss‚4.578154294591554 /loss/FRt2/assignment_loss‚4.502057552996261 0loss/FBt1/assignment_loss‚0.2066004705254091 0loss/SRqqt2/detection_loss‚3.654343851468879·hAzA‚(:Ï µäöÒäÝ©Þ& disk.nvme1c1n1.out‚228.90869140625 disk.rbd1.out‚0 disk.dm-1.in‚0$ disk./.usageGB‚1167.049186706543 disk.nvme0n1.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚574.42529296875 disk.nvme0c0n1.in‚0$ memory_percent‚95.76451778411865$ disk.nvme0n1.out‚368.51123046875$ disk.nvme1n1.out‚228.90869140625 disk.md0.in‚0 disk.md0.out‚9.546875 disk.dm-1.out‚ 2.52587890625& disk.nvme0c0n1.out‚368.51123046875( proc.memory.availableMB‚ 693.94140625 cpu‚25.625449610707935 network.sent‚417416) disk./.usagePercent‚76.01688237788913 disk.rbd1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2820.36328125* proc.memory.percent‚17.214131355285645 proc.cpu.threads‚17 network.recv‚126423 disk.rbd0.out‚0"’ disk.nvme1c1n1.in‚0hI&“ž‚(:ˆ µäöÒ¬ÂëÞ gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚261.339' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚87.113 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷I|×VŸ‚(:Ë ÄäöÒÄ úÝ disk.dm-0.out‚575.3134765625 disk.dm-1.out‚ 5.77978515625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚232.61572265625% proc.memory.availableMB‚ 693.28125 proc.cpu.threads‚17( disk./.usagePercent‚76.0168667240264% disk.nvme0c0n1.out‚369.7353515625& disk.nvme1c1n1.out‚232.61572265625 disk.rbd1.out‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2820.36328125* proc.memory.percent‚17.214131355285645 cpu‚25.617530104411685 network.sent‚450544 network.recv‚130542 disk.md0.in‚0 disk.md0.out‚ 9.91015625 disk.rbd1.in‚0$ memory_percent‚95.76854705810547% disk./.usageGB‚1167.0489463806152 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚369.7353515625 disk.rbd0.in‚0 disk.dm-0.in‚0àý|ž ‚(:“ ÄäöÒ›âËÞ gpu.0.gpu‚56.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.172' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.39066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µz?N¡‚(:à ÓäöÒâúŽÞ$ disk.nvme0c0n1.out‚ 412.251953125 disk.nvme0n1.in‚0 disk.rbd0.in‚0 network.recv‚134300) disk./.usagePercent‚76.14739509188688 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 234.806640625 disk.dm-0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.214131355285645 cpu‚25.60950196940231 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚95.76728343963623 proc.cpu.threads‚17 network.sent‚455783 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 5.91796875" disk.nvme0n1.out‚ 412.251953125( proc.memory.availableMB‚ 693.48828125# proc.memory.rssMB‚ 2820.36328125% disk./.usageGB‚1169.0528831481934$ disk.nvme1c1n1.out‚ 234.806640625 disk.md0.in‚0 disk.md0.out‚ 10.2421875 disk.dm-0.out‚ 619.16015625rD¢‚(:’ ÓäöÒâÓéÞ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚254.86' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.95333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øs•0£+ _wandbruntime‚362  _runtime‚362T}xsŤ¿ ' lr-AdamW/pg1‚0.00017833344711331434 ' lr-AdamW/pg2‚0.00017833344711331434 trainer/global_step‚399 $ _timestamp‚1.784525397894255e+09 _runtime‚ 362.164236869 _step‚28yk´É¥Ã trainer/global_step‚399 $ _timestamp‚1.784525397894255e+09 _runtime‚ 362.164236869 _step‚28 ' lr-AdamW/pg1‚0.00017833344711331434 ' lr-AdamW/pg2‚0.00017833344711331434hSï‚0¦+ _wandbruntime‚362  _runtime‚362›¨ ¬´'§®O YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59790668510720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3121105 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59345375905280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3121103 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59007830123376 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3884508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13848770750976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105448960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚928662 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 444605118822 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 448308685312 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59452418202838 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 448414134272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 445633849856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59790668510720 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13848646689792 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6995819 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59345392944640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚928957 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚928662 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 444605118822 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 448308685312 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59452418202838 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 448414134272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 445633849856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59790668510720 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13848646689792¸2Ûº0©+ _wandbruntime‚362  _runtime‚362•¹¯‰"ª¸M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1631914 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3894178 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59940411402752 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59940411402752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3894178 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 449429174784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 362.800815275 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13433536886784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59600957258562 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3884372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3884372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3128896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3128896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59601314857780 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59493667446784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7845253985311327e+09 B6DeviceStatsMonitor.on_train_batch_end/õ\È"5active.all.freed‚7013268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13433518274560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3128898 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 449532519424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103344640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9312463 kgÂ&«¼M TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚700950 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59940053111808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 445713507074 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59940411402752 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59940411402752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3894178 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 449429174784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 362.800815275 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13433536886784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59600957258562 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3884372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3884372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3128896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3128896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59601314857780 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59493667446784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7845253985311327e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7013268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13433518274560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3128898 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 449532519424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103344640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚931246 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1631914 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3894178 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B\¡[HÅbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPgS»sJÆbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¨I)èHÇbC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP Ô6JÈbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP^`BJÉbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPínÐHÊbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÿIÀJËbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP 4"JÌbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP.‚PNÍbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÂëéxJÎbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP¿yÌHÏbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPp‹jHÐbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP-ÑûoHÑbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP³IîHÒbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÔIÙ”JÓbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPŸj:¡JÔbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPòY7NÕbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP ú2¼JÖbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP“ÔÖzH×bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPx>Ú©HØbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPB étHÙbC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPϪ;Úb6 jet/accuracy_3_of_5"trainer/global_step2:JPÖW¯)NÛbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP?gñHÜbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPgÀÄ@Ýb; particle/accuracy_1_of_3"trainer/global_step2:JP’‹…HÞbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPÓ.‰,NßbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÀ|î.JàbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÁmêÔ@¤b; particle/accuracy_2_of_4"trainer/global_step2:JP=?ãÄJ¥bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÊ2²]J¦bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPØe°Ú@§b; particle/accuracy_2_of_3"trainer/global_step2:JPìæÅ¥H¨bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPo¬æŸ@©b; particle/accuracy_3_of_5"trainer/global_step2:JP€ÌÐ.JªbE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPŠÈÆ;«b6 jet/accuracy_2_of_5"trainer/global_step2:JP**ÅJ¬bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP±z_‘N­bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP«üÉN®bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPF`n;¯b6 jet/accuracy_4_of_4"trainer/global_step2:JPŽÌêJ°bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPØtZÜJ±bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPO ú¶J²bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP—'Î.J³bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÈ<9H´bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP‰õ¯NµbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPì—r©H¶bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP ØJ·bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP`õ¿ªJ¸bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP7 ¯èJ¹bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPOŠ„JºbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPû‚q»J»bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPË.,–;¼b6 jet/accuracy_1_of_3"trainer/global_step2:JPw¬V"H½bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPså»OH¾bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPå4J¿bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPR¾…~@Àb; particle/accuracy_1_of_2"trainer/global_step2:JPþ ÝGJÁbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPßV¹@JÂbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPðhJÃbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPG6ÀJÄbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP…==HÅbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP¼Y…X@Æb; particle/accuracy_5_of_5"trainer/global_step2:JPÝ@öHÇbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¥¥ÃoHÈbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÕî­ÚJÉbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP(NÊbI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPúxTõJËbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP±µ„sNÌbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¦ÂòJÍbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP! ×HÎbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP (JÏbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP©?®NÐbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPöЮ HÑbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP5ðºÈNÒbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP4i1JÓbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP">ÆÏNÔbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPvS¸ì@Õb; particle/accuracy_4_of_5"trainer/global_step2:JPcF«þNÖbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP]VùN×bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP/t*MHØbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÞ§„9JÙbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPõùJÚbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP5‰¨<Ûb7 particle/specificity"trainer/global_step2:JP?Ì&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9654761552810669 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00272654858417809 'particle/f_score‚0.5143603086471558 7validation_average_jet_accuracy‚0.11727450042963028 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 jet/accuracy_5_of_5‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05067947879433632 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.02777777798473835 ,jet/accuracy_2_of_2‚0.045707277953624725 epoch‚0 trainer/global_step‚399 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7631587982177734 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2109375 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.0922524631023407 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.028501613065600395 *jet/accuracy_1_of_4‚0.5516453981399536 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7016029953956604 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.0852985754609108 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.020988691598176956 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.09944922477006912 +jet/accuracy_2_of_3‚0.11365262418985367 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.027765518054366112 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6556347608566284 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9860358238220215 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 particle/accuracy_3_of_4‚1 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.029939357191324234 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9707110524177551 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.11959052085876465 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 (particle/accuracy‚0.7681323289871216 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.029145976528525352 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1726638823747635 /particle/accuracy_4_of_4‚0.9890753626823425 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9840185046195984 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1951805055141449 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.07976211607456207 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.028501613065600395 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9990025758743286 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0019654210191220045 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9298741221427917 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10812538117170334 jet/accuracy_2_of_5‚0 % _timestamp‚1.7845254157246096e+09 _step‚32 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.12928923964500427 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 jet/accuracy_4_of_4‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10079054534435272 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959821105003357 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.10490018874406815 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1733853965997696 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921271204948425 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.12203165143728256 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.10410181432962418 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.10470422357320786 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.053817085921764374 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.5195406079292297 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9774860143661499 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.986060619354248 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16746385395526886 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 _runtime‚ 379.993245568 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.23379629850387573 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.11712375283241272 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6666666865348816 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07230765372514725 particle/accuracy_5_of_5‚0 4 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9766845703125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47828033566474915 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8749333620071411 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07230765372514725 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.001997399376705289 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_4_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15764792263507843 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19511334598064423 +particle/specificity‚0.8218243718147278 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.0705249160528183 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.12034420669078827 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.1111111119389534 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.5717970132827759 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.07317809015512466 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1552993506193161 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.001736385514959693 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5518643260002136 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.428232878446579 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3812428414821625 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.19511334598064423 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9968254566192627 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.760816216468811 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.13748778402805328 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.021952640265226364 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.02958074025809765 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1965099722146988 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.07513097673654556 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9945015907287598 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10048030316829681 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959821105003357 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.3245123326778412 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.0627639889717102 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14371006190776825 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.083867646753788 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.0416666641831398 +jet/accuracy_1_of_1‚0.08853965997695923 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7115384340286255 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 ) Purity/*FRt1SRqqt2FBt/FBt_purity‚0.75 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9649369120597839 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993439257144928 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.09293273091316223 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.021934935823082924 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07953924685716629 ,jet/accuracy_3_of_3‚0.003678869688883424 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09975352883338928 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9832133054733276 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14662936329841614 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0338541679084301 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.10000000149011612 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.10325413942337036 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.15159612894058228 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437º6o2ÓÚz 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.028501613065600395 *jet/accuracy_1_of_4‚0.5516453981399536 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7016029953956604 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.0852985754609108 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.020988691598176956 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.09944922477006912 +jet/accuracy_2_of_3‚0.11365262418985367 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.027765518054366112 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6556347608566284 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9860358238220215 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 particle/accuracy_3_of_4‚1 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.029939357191324234 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9707110524177551 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.11959052085876465 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 (particle/accuracy‚0.7681323289871216 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.029145976528525352 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1726638823747635 /particle/accuracy_4_of_4‚0.9890753626823425 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9840185046195984 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1951805055141449 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.07976211607456207 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.028501613065600395 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9990025758743286 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0019654210191220045 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9298741221427917 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10812538117170334 jet/accuracy_2_of_5‚0 % _timestamp‚1.7845254157246096e+09 _step‚32 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.12928923964500427 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 jet/accuracy_4_of_4‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10079054534435272 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959821105003357 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.10490018874406815 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1733853965997696 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921271204948425 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.12203165143728256 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.10410181432962418 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.10470422357320786 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.053817085921764374 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.5195406079292297 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9774860143661499 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.986060619354248 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16746385395526886 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 _runtime‚ 379.993245568 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.23379629850387573 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.11712375283241272 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6666666865348816 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07230765372514725 particle/accuracy_5_of_5‚0 4 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9766845703125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47828033566474915 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8749333620071411 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07230765372514725 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.001997399376705289 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_4_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15764792263507843 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19511334598064423 +particle/specificity‚0.8218243718147278 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.0705249160528183 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.12034420669078827 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.1111111119389534 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.5717970132827759 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.07317809015512466 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1552993506193161 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.001736385514959693 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5518643260002136 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.428232878446579 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3812428414821625 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.19511334598064423 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9968254566192627 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.760816216468811 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.13748778402805328 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.021952640265226364 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.02958074025809765 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1965099722146988 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.07513097673654556 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9945015907287598 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10048030316829681 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959821105003357 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.3245123326778412 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.0627639889717102 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14371006190776825 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.083867646753788 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.0416666641831398 +jet/accuracy_1_of_1‚0.08853965997695923 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7115384340286255 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 ) Purity/*FRt1SRqqt2FBt/FBt_purity‚0.75 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9649369120597839 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993439257144928 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.09293273091316223 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.021934935823082924 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07953924685716629 ,jet/accuracy_3_of_3‚0.003678869688883424 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09975352883338928 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9832133054733276 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14662936329841614 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0338541679084301 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.10000000149011612 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.10325413942337036 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.15159612894058228 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.07621876150369644 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.09442032873630524 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.03941953927278519 ;"Purity/*FRt1SRqqt0FBt/event_purity‚0.060200341045856476 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.10470422357320786 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11595210433006287 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.18372824788093567 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_4‚0 particle/accuracy_1_of_1‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9834421873092651 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11446893960237503 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *loss/val_total_loss‚25.820184612942274 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.28703704476356506 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.18447625637054443 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.06713254004716873 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.1343701034784317 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.08699925243854523 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06105079501867294 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.28703704476356506 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964722394943237 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9887032508850098 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.169499933719635 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.00406578229740262 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.02726712077856064 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.36125022172927856 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4917547106742859 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9548121094703674 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.21921123564243317 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9969338178634644 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.05733181908726692 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4077827036380768 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06161242350935936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11253059655427933 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.1198303326>„ú´ñ 9643784 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955227375030518 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06276806443929672 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.047874145209789276 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.02074848860502243 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.12522728741168976 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.09395314753055573 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9519966840744019 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.14146389067173004 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.04697896167635918 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.019841274246573448 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.0837433710694313 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004123642574995756 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.06355200707912445 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9654761552810669 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00272654858417809 'particle/f_score‚0.5143603086471558 7validation_average_jet_accuracy‚0.11727450042963028 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 jet/accuracy_5_of_5‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05067947879433632 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.02777777798473835 ,jet/accuracy_2_of_2‚0.045707277953624725 epoch‚0 trainer/global_step‚399 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7631587982177734 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2109375 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.0922524631023407 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 (®nLWÔ‚(:Ì ñäöÒÕ¨‡Þ* proc.memory.percent‚17.434310913085938 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme1c1n1.out‚268.4716796875 disk.dm-0.in‚0' proc.memory.availableMB‚ 494.5546875 network.sent‚508326 network.recv‚142959 disk.rbd0.out‚0 disk.rbd1.out‚0 disk.nvme0n1.in‚0$ memory_percent‚96.98147773742676 proc.memory.rssMB‚ 2856.4375 cpu‚25.7121638826104) disk./.usagePercent‚76.19671320763808 disk.nvme1n1.in‚0# disk.nvme1n1.out‚268.4716796875 disk.md0.in‚0 disk.md0.out‚ 11.18359375" disk.dm-0.out‚2504.26123046875 disk.dm-1.in‚0 disk.dm-1.out‚ 38.283203125% disk./.usageGB‚1169.8100395202637 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2298.05126953125% disk.nvme0n1.out‚2298.05126953125¡êšjžÕ‚(:“ ñäöÒ½ÿ„ß gpu.0.gpu‚38.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚285.913' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.30433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïü€VÖ‚(:Ë €åöÒ¾Û‹Þ cpu‚25.70045345064669 network.recv‚147753% disk./.usageGB‚1169.7637634277344# disk.nvme0n1.out‚2870.466796875 disk.rbd0.out‚0% disk.nvme1c1n1.out‚1024.962890625 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.dm-0.out‚3373.751953125 network.sent‚567288) disk./.usagePercent‚76.19369896940135 disk.nvme1c1n1.in‚0 disk.md0.in‚0" proc.memory.rssMB‚ 2856.4453125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚2870.466796875# disk.nvme1n1.out‚1024.962890625 disk.md0.out‚ 11.48046875 disk.dm-0.in‚0$ memory_percent‚96.98319435119629 disk.dm-1.in‚0 disk.dm-1.out‚ 497.060546875 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 494.2734375* proc.memory.percent‚17.434358596801758358>“ׂ(:ˆ €åöÒÌ€ÚÞ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚264.531' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.177 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô54PØ‚(:Å åöÒ³€Þ network.sent‚570582 network.recv‚151491 disk.md0.in‚0 disk.dm-1.out‚ 497.18359375 disk.rbd0.in‚0$ memory_percent‚96.98333740234375& disk.nvme0c0n1.out‚2881.2822265625$ disk./.usageGB‚1169.763687133789" disk.nvme1n1.out‚ 1027.23828125 disk.dm-0.in‚0 disk.nvme0n1.in‚0$ disk.nvme1c1n1.out‚ 1027.23828125 disk.rbd1.out‚0 cpu‚25.707896738242532 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚2881.2822265625 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0" proc.memory.availableMB‚494.25! proc.memory.rssMB‚ 2856.453125* proc.memory.percent‚17.434406280517578 proc.cpu.threads‚17) disk./.usagePercent‚76.19369399992112 disk.nvme1n1.in‚0 disk.md0.out‚ 11.83203125! disk.dm-0.out‚3385.9462890625 disk.dm-1.in‚0 disk.rbd0.out‚0± TžÙ‚(:“ åöÒ”¬ÅÞ gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚276.941' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.31366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¯0Ú+ _wandbruntime‚425  _runtime‚425IKÛÅÛ¿ ' lr-AdamW/pg2‚0.00017877764110431425 trainer/global_step‚449 % _timestamp‚1.7845254611608422e+09 _runtime‚ 425.42969259 _step‚33 ' lr-AdamW/pg1‚0.00017877764110431425Zƒo$ÉÜà trainer/global_step‚449 % _timestamp‚1.7845254611608422e+09 _runtime‚ 425.42969259 _step‚33 ' lr-AdamW/pg1‚0.00017877764110431425 ' lr-AdamW/pg2‚0.00017877764110431425!E¹j0Ý+ _wandbruntime‚425  _runtime‚4252ßÏ{¸'Þ²O P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4440573 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69222624283136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68300602465840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 521520818176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚790029 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1084306 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69222624283136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68701444888064 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1874610 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8084277 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 524886349824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 521179395072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 521179395072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 521520818176 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3633561 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1084578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111299584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 520392179967 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3633561 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69222982745600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚34 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92687360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3633563 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 524979037184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68701444888064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1874335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68820994645807 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15081930170880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68300619505200“9‡G¼'߶O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8074134 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68701461927424 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8084277 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 524886349824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 521179395072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 521179395072 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 521520818176 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3633561 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1084578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111299584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 520392179967 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3633561 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69222982745600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚34 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92687360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3633563 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 524979037184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68701444888064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1874335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68820994645807 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15081930170880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68300619505200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4440573 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69222624283136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68300602465840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 521520818176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚790029 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1084306 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69222624283136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68701444888064 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1874610 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3641356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69372008884224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 % _timestamp‚1.7845254617747636e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4460384 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4450229 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68448033133312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1878317 trainer/global_step‚449 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68849719390208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_=Êi{batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 522289494016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15115129053184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 521500568219 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3641356 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3641354 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8101740 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68849736429568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 522289494016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1878593 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8091583 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69372367353856 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1878317 trainer/global_step‚449 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68849719390208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 522289494016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15115129053184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 521500568219 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3641356 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3641354 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8101740 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68849736429568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 522289494016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1878593 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8091583 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69372367353856 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3641356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69372008884224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 % _timestamp‚1.7845254617747636e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4460384 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4450229 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68448033133312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107367424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4023213 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979648 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76248368833441 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2076116 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚876732 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76691918492160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 576748496896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76692276954624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76115169995264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4023211 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8948436 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚ 468.599330524 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 577089920000 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 577089920000 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8958579 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4925225 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75672135839440?ìÄc€ñ¸O JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75672152878800 _step‚38 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76692276954624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76115187034624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4023213 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76248011232815 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4925225 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75672135839440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17321847963648 =6DeviceStatsMonitor.on_train_batch_start/max_split¼ˆ´Ó> _size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 580885676544 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 576215954641 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76115187034624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76115169995264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚499 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16740943674880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4925225 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1199114 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2075843 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 % _timestamp‚1.7845255043313558e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16740962287104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4935366 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 575875393375 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107367424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4023213 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979648 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76248368833441 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2076116 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚876732 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76691918492160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 576748496896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76692276954624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76115169995264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4023211 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8948436 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚ 468.599330524 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 577089920000 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 577089920000 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8958579 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76263461536768 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8965885 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2079777 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚878463 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76263444497408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4934881 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1201595 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 581893774848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4031004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845255049456997e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 578200026112 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 582005329408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 577324343005 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8976042 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4031004 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17356017719808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76263444497408 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17356147886592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111554560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76396907889277 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚ 469.213965825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75819566506912 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 577858595840 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚878466 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 577858595840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 578200026112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130166784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16774142557184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16774123944960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚499 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17356147886592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111554560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76396907889277 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚ 469.213965825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75819566506912 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 577858595840 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚878466 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 577858595840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 578200026112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130166784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16774142557184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16774123944960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚499 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76263461536768 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8965885 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2079777 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚878463 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76263444497408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4934881 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1201595 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 581893774848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4031004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845255049456997e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 578200026112 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 582005329408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 577324343005 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8976042 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4031004 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17356017719808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76263444497408 F0ÆÿÀ % _timestamp‚1.7845255476477647e+09 _runtime‚ 511.916480546 _step‚41 ' lr-AdamW/pg1‚0.00017875490637297243 ' lr-AdamW/pg2‚0.00017875490637297243 trainer/global_step‚549’ù'áÊ€Ä ' lr-AdamW/pg1‚0.00017875490637297243 ' lr-AdamW/pg2‚0.00017875490637297243 trainer/global_step‚549 % _timestamp‚1.7845255476477647e+09 _runtime‚ 511.916480546 _step‚41)•ZÎ80+ _wandbruntime‚511  _runtime‚511˜yd¸'‚²O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 632659021824 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19036609901056 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 636652722176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18399957178880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83043686252400 trainer/global_step‚549 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83528912141824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 631699168049 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84161212701184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115493888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚42 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1313504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5409877 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 632317598720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83675027819823 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784525547648477e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19036725394944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5420018 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18399975791104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84161212701184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2276648 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚963432 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 632317598720 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5409877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9832881 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 511.916740216 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84161571163648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83528895102464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 636749603840 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4412863 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1313219 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 632659021824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 511.916740216 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84161571163648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83528895102464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 636749603840 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4412863 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1313219 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 632659021824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 632659021824 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19036609901056 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 636652722176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18399957178880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83043686252400 trainer/global_step‚549 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83528912141824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 631699168049 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84161212701184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115493888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚42 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1313504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5409877 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 632317598720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83675027819823 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784525547648477e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19036725394944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5420018 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18399975791104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84161212701184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2276648 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚963432*6­}0„+ _wandbruntime‚512  _runtime‚512üÌé>Ã&…½M K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 trainer/global_step‚549 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9840187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5419533 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637769668608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 633769127936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83677169604608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2280913 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1315747 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 632466995035 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83191116919872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98971648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9850344 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 633427697664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 633427697664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18433137448960 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 637868640256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83191099880512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117583872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845255482675097e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84310955771904 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4420654 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5429688 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚43 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1315478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 512.535825252 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84310955771904 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19071024701440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18433156061184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5419533 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚965166 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83677169604608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2280641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4420654 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 632807556413 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83677186643968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84310597302272 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 633769127936 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2280913 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1315747 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 632466995035 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83191116919872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytež¬¬eÁ"s.small_pool.current‚98971648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9850344 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 633427697664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 633427697664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18433137448960 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 637868640256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83191099880512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117583872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845255482675097e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84310955771904 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4420654 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5429688 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚43 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1315478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 512.535825252 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84310955771904 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19071024701440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18433156061184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5419533 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚965166 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83677169604608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2280641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4420654 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 632807556413 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83677186643968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84310597302272 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 633769127936 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5419533 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637769668608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 633769127936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83677169604608+™°ö®0‡+ _wandbruntime‚512  _runtime‚512͹üüˆö 2loss/SRqqt1/assignment_loss‚1.4014786740825846 1loss/FBt2/assignment_loss‚0.41676542128450156 /loss/FRt2/detection_loss‚0.5442489569353408 _step‚44 0loss/FRt1/assignment_loss‚2.6338042928432595 0loss/FBt1/assignment_loss‚0.1277573452212046 trainer/global_step‚549 $ _timestamp‚1.784525548269232e+09 _runtime‚ 512.537099834 2loss/SRqqt2/assignment_loss‚1.5667300276985072 /loss/FRt1/detection_loss‚1.3725968598534257 1loss/SRqqt1/detection_loss‚0.8371466107381351 0loss/SRqqt2/detection_loss‚2.313573053646881 -loss/FBt2/detection_loss‚7.27326735380021 %loss/total_loss‚25.03461483059482 .loss/FRt2/assignment_loss‚3.97277460756418 .loss/FBt1/detection_loss‚2.574471626926589 epoch‚1æ#‡‰ú 0loss/FRt1/assignment_loss‚2.6338042928432595 0loss/FBt1/assignment_loss‚0.1277573452212046 trainer/global_step‚549 $ _timestamp‚1.784525548269232e+09 _runtime‚ 512.537099834 2loss/SRqqt2/assignment_loss‚1.5667300276985072 /loss/FRt1/detection_loss‚1.3725968598534257 1loss/SRqqt1/detection_loss‚0.8371466107381351 0loss/SRqqt2/detection_loss‚2.313573053646881 -loss/FBt2/detection_loss‚7.27326735380021 %loss/total_loss‚25.03461483059482 .loss/FRt2/assignment_loss‚3.97277460756418 .loss/FBt1/detection_loss‚2.574471626926589 epoch‚1 2loss/SRqqt1/assignment_loss‚1.4014786740825846 1loss/FBt2/assignment_loss‚0.41676542128450156 /loss/FRt2/detection_loss‚0.5442489569353408 _step‚44,Ô!¨ýZŠ‚(:Ï øåöÒºÍÂÞ( proc.memory.percent‚17.4344539642334 proc.cpu.threads‚17 network.sent‚734826 network.recv‚179296% disk./.usageGB‚1169.7654037475586 disk.dm-1.out‚ 499.76953125 disk.nvme0n1.in‚0# disk.nvme1n1.out‚1032.966796875" proc.memory.rssMB‚ 2856.4609375 cpu‚25.67075530831151) disk./.usagePercent‚76.19380581322642 disk.md0.in‚0" disk.dm-0.out‚3394.78466796875' disk.nvme0c0n1.out‚2891.11474609375 disk.rbd0.out‚0% disk.nvme1c1n1.out‚1032.966796875 disk.dm-0.in‚0% disk.nvme0n1.out‚2891.11474609375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 13.71484375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd1.out‚0$ memory_percent‚96.98736667633057( proc.memory.availableMB‚ 493.58984375c^óž‹‚(:“ øåöÒðà‡ß gpu.0.gpu‚58.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.331' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.11033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N`YŒ‚(:Î ‡æöÒ‹üÞ disk.dm-0.in‚0% disk.nvme0n1.out‚2893.24267578125& proc.memory.availableMB‚ 493.828125" proc.memory.rssMB‚ 2856.4609375& disk.nvme1c1n1.out‚1033.4853515625 disk.rbd1.in‚0 disk.rbd1.out‚0" disk.dm-0.out‚3396.83154296875 disk.rbd0.out‚0$ memory_percent‚96.98591232299805( proc.memory.percent‚17.4344539642334 disk.nvme1c1n1.in‚0 disk.md0.out‚ 13.953125 disk.dm-1.out‚ 499.83984375 cpu‚25.665292767592643 proc.cpu.threads‚17% disk./.usageGB‚1169.7657585144043 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2893.24267578125 disk.nvme0n1.in‚0 disk.rbd0.in‚0 network.sent‚783334 network.recv‚183961) disk./.usagePercent‚76.19382892130952$ disk.nvme1n1.out‚1033.4853515625š²·ž‚(:“ ‡æöÒ´ÞàÞ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.939' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.64633333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-l›d0Ž+ _wandbruntime‚554  _runtime‚554¯­ÞÜÆÀ ' lr-AdamW/pg2‚0.00017873492895783226 trainer/global_step‚599 % _timestamp‚1.7845255903883903e+09 _runtime‚ 554.656519021 _step‚45 ' lr-AdamW/pg1‚0.00017873492895783226 âLÔÊÄ _step‚45 ' lr-AdamW/pg1‚0.00017873492895783226 ' lr-AdamW/pg2‚0.00017873492895783226 trainer/global_step‚599 % _timestamp‚1.7845255903883903e+09 _runtime‚ 554.656519021-褊}0‘+ _wandbruntime‚554  _runtime‚554gbžÀ'’ºO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1050132 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10707183 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10697040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1427638 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90415202586640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5904670 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20751574167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 _step‚46 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90942637249024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130173952 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91630506910208 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 687886700544 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 688228123648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 687886700544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 692473310720 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 686841820191 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4802511 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2477767 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1050129 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90942637249024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5904670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚ 554.657276802 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4802513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91102044406831 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91630865372672 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1427638 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90415202586640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5904670 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20751574167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 _step‚46 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90942637249024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130173952 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91630506910208 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 687886700544 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 688228123648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 687886700544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 692473310720 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 686841820191 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4802511 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2477767 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1050129 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90942637249024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5904670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚ 554.657276802 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4802513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91102044406831 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91630865372672 IDeviceSts‹²atsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91780249980928 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5914340 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10714489 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20785868579328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91090911751168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 689338229760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2481736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91250583462555 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91250941063293 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91090911751168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5914340 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1051866 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5914340 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1051866 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91780249980928 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5914340 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10714489 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20785868579328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91090911751168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 689338229760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2481736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91250583462555 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488/ŒCE 0™+ _wandbruntime‚555  _runtime‚555ZÁvºÿšù /loss/FRt1/assignment_loss‚2.501341477383015 0loss/FBt2/assignment_loss‚0.4645853470770064 1loss/SRqqt1/detection_loss‚0.8212855799884426 .loss/FBt2/detection_loss‚5.744072584242656 &loss/total_loss‚23.997394318919923 2loss/SRqqt2/assignment_loss‚1.6336161008743693 1loss/FBt1/assignment_loss‚0.13964668380052692 1loss/SRqqt2/detection_loss‚2.6363140701664483 /loss/FBt1/detection_loss‚2.8965536057531813 trainer/global_step‚599 _runtime‚ 555.275687659 1loss/SRqqt1/assignment_loss‚1.300774674114284 .loss/FRt2/detection_loss‚0.502832141303808 % _timestamp‚1.7845255910078743e+09 _step‚48 /loss/FRt2/assignment_loss‚3.977467868477106 /loss/FRt1/detection_loss‚1.3789041857390805 epoch‚1à žm›ý /loss/FRt1/detection_loss‚1.3789041857390805 epoch‚1 /loss/FRt1/assignment_loss‚2.501341477383015 0loss/FBt2/assignment_loss‚0.4645853470770064 1loss/SRqqt1/detection_loss‚0.8212855799884426 .loss/FBt2/detection_loss‚5.744072584242656 &loss/total_loss‚23.997394318919923 2loss/SRqqt2/assignment_loss‚1.6336161008743693 1loss/FBt1/assignment_loss‚0.13964668380052692 1loss/SRqqt2/detection_loss‚2.6363140701664483 /loss/FBt1/detection_loss‚2.8965536057531813 trainer/global_step‚599 _runtime‚ 555.275687659 1loss/SRqqt1/assignment_loss‚1.300774674114284 .loss/FRt2/detection_loss‚0.502832141303808 % _timestamp‚1.7845255910078743e+09 _step‚48 /loss/FRt2/assignment_loss‚3.9774678684771060ðÖc8^œ‚(:Ó ¥æöÒ™ŒûÝ network.recv‚192656( disk./.usagePercent‚76.1938224609852 disk.md0.in‚0 disk.md0.out‚ 14.5546875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2896.52197265625% disk.nvme0n1.out‚2896.52197265625% disk./.usageGB‚1169.7656593322754 disk.rbd1.out‚0% disk.nvme1n1.out‚1034.68408203125 disk.nvme0n1.in‚0 disk.rbd0.out‚0 cpu‚25.6555703905371 network.sent‚840457' disk.nvme1c1n1.out‚1034.68408203125( proc.memory.availableMB‚ 493.75390625" proc.memory.rssMB‚ 2856.4609375( proc.memory.percent‚17.4344539642334 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚3399.81982421875! disk.dm-1.out‚500.01318359375 disk.rbd0.in‚0$ memory_percent‚96.98636531829834ú «ž‚(:“ ¥æöÒÄ‘äÞ gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.474' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.49133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hú`蛞‚(: ´æöÒ©¹·Þ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.41' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.13666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0܉QTŸ‚(:É ´æöÒÉïŽß$ memory_percent‚96.98638916015625 network.recv‚197312 disk.nvme0n1.in‚0# disk.nvme0n1.out‚2898.087890625 disk.rbd0.in‚0" proc.memory.availableMB‚493.75) proc.memory.percent‚17.43447780609131 cpu‚25.65033951715052 proc.cpu.threads‚17 network.sent‚894737$ disk./.usageGB‚1169.766040802002! disk.dm-1.out‚500.08740234375% disk.nvme0c0n1.out‚2898.087890625 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0) disk./.usagePercent‚76.19384730838638 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 14.953125" disk.dm-0.out‚3401.00048828125# proc.memory.rssMB‚ 2856.46484375 disk.rbd0.out‚0& disk.nvme1c1n1.out‚1035.2294921875$ disk.nvme1n1.out‚1035.2294921875FñA20 + _wandbruntime‚597  _runtime‚597"j/QÅ¡¿ _runtime‚ 597.731633888 _step‚49 ' lr-AdamW/pg1‚0.00017870921309764466 ' lr-AdamW/pg2‚0.00017870921309764466 trainer/global_step‚649 $ _timestamp‚1.784525633463501e+09ý}±†É¢Ã _runtime‚ 597.731633888 _step‚49 ' lr-AdamW/pg1‚0.00017870921309764466 ' lr-AdamW/pg2‚0.00017870921309764466 trainer/global_step‚649 $ _timestamp‚1.784525633463501e+091éìʉ0£+ _wandbruntime‚597  _runtime‚597¡…<À¤ºO TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98529060993839 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 743797225472 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 743455802368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5192163 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98529418594465 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98356362356224 % _timestamp‚1.7845256334645042e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5192163 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21718002799104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1136829 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98356345316864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1542165 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21717984186880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99099801119232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6389322 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98356362356224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97786735960240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99100159581696 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99099801119232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚ 597.736411046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 748334766080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚50 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6379181 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2678698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101075968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119688192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312ÿˆË Ä'¥¾O JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 743797225472 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 743455802368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5192163 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98529418594465 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98356362356224 % _timestamp‚1.7845256334645042e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5192163 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21718002799104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1136829 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98356345316864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1542165 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21717984186880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99099801119232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6389322 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98356362356224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97786735960240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99100159581696 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99099801119232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚ 597.736411046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 748334766080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚50 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6379181 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2678698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101075968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119688192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚985290609938392GM0¦+ _wandbruntime‚598  _runtime‚598ÄSÏÊ&§ÄM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1138566 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22500609622016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22500741885952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99249544189952 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5199956 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 743433421851 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5199954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11588791 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 744907331584 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6388837 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98504619819008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6398992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21751183069184 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99249185720320 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2682676 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97934166627712 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5199956 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1138563 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1544113 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845256340804179e+09 _step‚51 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98504636858368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98677600049563 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98504619819008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 749445165056 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 743773983229 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚649 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11588791 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6398992 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6388837 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99249544189952 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 744907331584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98677957650301 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1544393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 744565901312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 744565901312 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 743433421851 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5199954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11588791 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 744907331584 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6388837 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98504619819008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6398992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21751183069184 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99249185720320 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2682676 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97934166627712 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5199956 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1138563 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1544113 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845256340804179e+09 _step‚51 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98504636858368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98677600049563 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98504619819008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 749445165056 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 743773983229 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚649 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11588791 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6398992 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6388837 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99249544189952 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 744907331584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VCDeviceStatsMonitor.on_trainÓÌ Úk_batch_end/requested_bytes.all.allocated‚98677957650301 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1544393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 744565901312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 744565901312 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99249544189952 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51999563Ö&2u0©+ _wandbruntime‚598  _runtime‚598Œ’(gªú _runtime‚ 598.349681503 _step‚52 /loss/FBt1/detection_loss‚1.8876268216934278 epoch‚1 1loss/SRqqt1/assignment_loss‚1.150255019219821 1loss/FBt1/assignment_loss‚0.13283410808402019 0loss/FBt2/assignment_loss‚0.4607609097718412 trainer/global_step‚649 % _timestamp‚1.7845256340818722e+09 0loss/FRt2/assignment_loss‚3.9142124628657617 2loss/SRqqt2/assignment_loss‚1.5652817720845342 .loss/FRt1/detection_loss‚1.357773120393242 /loss/FRt2/detection_loss‚0.5435645064927914 1loss/SRqqt1/detection_loss‚0.8399205824614554 1loss/SRqqt2/detection_loss‚2.7631266478071166 /loss/FRt1/assignment_loss‚2.296983242034912 .loss/FBt2/detection_loss‚3.740812867467351 &loss/total_loss‚20.653152060376275áAŸ«þ trainer/global_step‚649 % _timestamp‚1.7845256340818722e+09 0loss/FRt2/assignment_loss‚3.9142124628657617 2loss/SRqqt2/assignment_loss‚1.5652817720845342 .loss/FRt1/detection_loss‚1.357773120393242 /loss/FRt2/detection_loss‚0.5435645064927914 1loss/SRqqt1/detection_loss‚0.8399205824614554 1loss/SRqqt2/detection_loss‚2.7631266478071166 /loss/FRt1/assignment_loss‚2.296983242034912 .loss/FBt2/detection_loss‚3.740812867467351 &loss/total_loss‚20.653152060376275 _runtime‚ 598.349681503 _step‚52 /loss/FBt1/detection_loss‚1.8876268216934278 epoch‚1 1loss/SRqqt1/assignment_loss‚1.150255019219821 1loss/FBt1/assignment_loss‚0.13283410808402019 0loss/FBt2/assignment_loss‚0.46076090977184124RŠ¾)Y¬‚(:Î ÃæöÒ£ŠÒÞ' disk.nvme1c1n1.out‚1036.59814453125 disk.nvme1n1.in‚0 disk.md0.out‚ 15.41796875 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚2902.759765625 disk.rbd0.out‚0) proc.memory.percent‚17.43447780609131 disk.rbd1.in‚0 disk.md0.in‚0 disk.dm-0.out‚3405.935546875 disk.nvme0n1.in‚0# disk.nvme0n1.out‚2902.759765625$ memory_percent‚96.98715209960938 cpu‚25.646318010829837 proc.cpu.threads‚17 network.recv‚202338% disk./.usageGB‚1169.7654991149902 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1036.59814453125 disk.dm-1.in‚0# proc.memory.availableMB‚493.625# proc.memory.rssMB‚ 2856.46484375) disk./.usagePercent‚76.19381202507671 disk.rbd1.out‚0 disk.dm-0.in‚0! disk.dm-1.out‚500.21142578125 disk.rbd0.in‚0 network.sent‚898640¼¿D?ž­‚(:“ ÃæöÒ÷û”ß gpu.0.gpu‚95.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.321' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.77366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'!Î$^®‚(:Ó ÒæöÒ»‰Þ disk.nvme1c1n1.in‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚1037.36083984375 network.sent‚961482$ disk.nvme0n1.out‚2903.9013671875$ memory_percent‚96.99199199676514) proc.memory.percent‚17.43447780609131 network.recv‚207535 disk.md0.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚1037.36083984375# proc.memory.rssMB‚ 2856.46484375 proc.cpu.threads‚17) disk./.usagePercent‚76.19380978881061% disk./.usageGB‚1169.7654647827148 disk.dm-0.out‚3406.587890625! disk.dm-1.out‚500.38330078125 disk.rbd1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 492.83203125 cpu‚25.650016464959553 disk.md0.out‚ 15.921875 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚2903.9013671875¨µ4Äž¯‚(:“ ÒæöÒ¢ëÝÞ gpu.0.gpu‚40.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.325' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.44166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÏõ_°‚(:Ô áæöÒ´¡‘Þ disk.dm-0.in‚0! disk.dm-0.out‚3407.8486328125! disk.dm-1.out‚500.55712890625 disk.nvme0c0n1.in‚0# disk.nvme0n1.out‚2905.353515625 disk.rbd0.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2856.46484375) disk./.usagePercent‚76.19381351592078 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1037.92138671875) proc.memory.percent‚17.43447780609131 disk.md0.out‚ 16.18359375( proc.memory.availableMB‚ 493.02734375 cpu‚25.660840419082216 proc.cpu.threads‚17 network.sent‚968249 network.recv‚211093% disk./.usageGB‚1169.7655220031738% disk.nvme0c0n1.out‚2905.353515625 disk.rbd0.out‚0' disk.nvme1c1n1.out‚1037.92138671875 disk.rbd1.out‚0$ memory_percent‚96.99079990386963Dp‡V±‚(:’ áæöÒ°ƒÖÞ gpu.0.gpu‚53.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚291.29' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.09666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0夎Ï0²+ _wandbruntime‚641  _runtime‚641]J¶ùƳÀ _runtime‚ 641.246033016 _step‚53 ' lr-AdamW/pg1‚0.00017867776044409118 ' lr-AdamW/pg2‚0.00017867776044409118 trainer/global_step‚699 % _timestamp‚1.7845256769779527e+09‰LzÊ´Ä _step‚53 ' lr-AdamW/pg1‚0.00017867776044409118 ' lr-AdamW/pg2‚0.00017867776044409118 trainer/global_step‚699 % _timestamp‚1.7845256769779527e+09 _runtime‚ 641.2460330165Ÿ’ 0µ+ _wandbruntime‚641  _runtime‚641ÈXhË'¶ÅO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24181266243072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5581813 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2879571 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 799024904192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105770070424064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106569095328256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106569095328256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105158286373200 trainer/global_step‚699 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6863833 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105956435181473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799366327296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6873974 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24181133971968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113658880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23377016303104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105956077580847 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106569453790720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1223532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12445644 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 799366327296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12455787 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1223529 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚54 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23376997690880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 $ _timestamp‚1.784525676978639e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6873974 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105770087463424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105158269333840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 804249939968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 799024904192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 641.246603817 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5581811 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 797808247007 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184s‘Ø£Ï'·ÉO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105956077580847 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106569453790720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1223532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12445644 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 799366327296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12455787 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1223529 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚54 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23376997690880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 $ _timestamp‚1.784525676978639e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6873974 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105770087463424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105158269333840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 804249939968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 799024904192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 641.246603817 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5581811 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 797808247007 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24181266243072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5581813 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2879571 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 799024904192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105770070424064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106569095328256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106569095328256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105158286373200 trainer/global_step‚699 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6863833 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105956435181473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799366327296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6873974 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24181133971968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113658880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚233770163031046…åVà0¸+ _wandbruntime‚641  _runtime‚641qN:¹ÎM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12473250 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106718838398976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225263 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6873489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12463093 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 805252768768 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2883561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 trainer/global_step‚699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6883644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105305717040672 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5589604 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6883644 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 # _timestamp‚1.78452567759712e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1658569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5589606 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106104616636571 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5589604 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23410196573184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109457408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105918344926208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStaŸiÿÔštsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106718479929344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _step‚55 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105918361965568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 798916635259 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24215558799360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 805362226176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106104974237309 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799257196637 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2883561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 trainer/global_step‚699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6883644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105305717040672 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5589604 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6883644 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 # _timestamp‚1.78452567759712e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1658569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5589606 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106104616636571 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5589604 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23410196573184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109457408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105918344926208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106718479929344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _step‚55 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105918361965568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 798916635259 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24215558799360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 805362226176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106104974237309 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799257196637 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 F“¿‚(:ˆ ðæöÒè•‘ß gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚282.837' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.279 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ñ‚gbÀ‚(:× ÿæöÒÊËÞ network.sent‚ 1.035364e+06) disk./.usagePercent‚76.19381127965468 disk.dm-1.in‚0! disk.dm-1.out‚500.83056640625' disk.nvme1c1n1.out‚1038.93408203125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1038.93408203125) proc.memory.percent‚17.43457317352295 disk.md0.out‚ 16.609375$ disk.nvme0n1.out‚2907.9951171875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚99.97506141662598% proc.memory.availableMB‚ 4.0859375 cpu‚25.677732846590846% disk./.usageGB‚1169.7654876708984 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2856.48046875 proc.cpu.threads‚17 network.recv‚219303 disk.md0.in‚0! disk.dm-0.out‚3410.2666015625& disk.nvme0c0n1.out‚2907.9951171875 disk.rbd0.out‚0 disk.rbd1.out‚0é…»žÁ‚(:“ ÿæöÒÇñÙÞ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.668' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.55600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÎMœÂ‚(:‘ ŽçöÒ˜Ž©Þ gpu.0.gpu‚9.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.072' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.35733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦îº•`Â(:Õ ŽçöÒÁçˆß disk.nvme1n1.in‚0 disk.dm-0.in‚0! disk.dm-0.out‚3410.8916015625# proc.memory.rssMB‚ 2856.48046875) proc.memory.percent‚17.43457317352295 cpu‚25.6828433832014) disk./.usagePercent‚76.19381898234904% disk./.usageGB‚1169.7656059265137% disk.nvme0c0n1.out‚2909.201171875 disk.rbd0.in‚0 disk.md0.out‚ 17.2421875$ memory_percent‚99.97656345367432 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.dm-1.out‚500.88916015625 disk.nvme0c0n1.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 3.83984375' disk.nvme1c1n1.out‚1039.75341796875 network.sent‚ 1.040688e+06 network.recv‚223355 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚1039.75341796875 disk.md0.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚2909.201171875CÍrµ0Ä+ _wandbruntime‚684  _runtime‚6842ï³ÃŽ _step‚57 & lr-AdamW/pg1‚0.0001786405730173171 & lr-AdamW/pg2‚0.0001786405730173171 trainer/global_step‚749 $ _timestamp‚1.784525720276905e+09 _runtime‚ 684.545023213*´åºÇÆÁ _runtime‚ 684.545023213 _step‚57 & lr-AdamW/pg1‚0.0001786405730173171 & lr-AdamW/pg2‚0.0001786405730173171 trainer/global_step‚749 $ _timestamp‚1.784525720276905e+099ìÐù0Ç+ _wandbruntime‚684  _runtime‚684ºóJÌ'ÈÆO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113183812570624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25895918420480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 859907225600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1310232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1770367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113383451768481 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 860020884480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113183795531264 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 854935429120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3080883 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113183812570624 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5971461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13330089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113383094167855 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113183795531264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5971461 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7358626 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112529819746800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 853291460415 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7358626 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5971463 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114038389537280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1310229 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7348485 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25036011194880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚58 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 854594006016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5971463 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114038747999744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114038747999744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25036029807104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112529802707440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1770651 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113658880 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 853632021681 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114038389537280 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25896050691584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3080596 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845257202778687e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13319946ÖoµäCÉÊO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114038747999744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114038747999744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25036029807104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112529802707440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1770651 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113658880 ]LDeviceStatsMonitor.on_train_batch_start/reqVR°#uested_bytes.small_pool.allocated‚ 853632021681 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114038389537280 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25896050691584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3080596 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845257202778687e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13319946 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113183812570624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25895918420480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 859907225600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1310232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1770367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113383451768481 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 860020884480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113183795531264 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 854935429120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3080883 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113183812570624 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5971461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13330089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113383094167855 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113183795531264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5971461 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7358626 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112529819746800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 853291460415 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7358626 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5971463 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114038389537280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1310229 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7348485 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25036011194880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚58 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 854594006016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5971463:8ÚD0Ê+ _wandbruntime‚685  _runtime‚685î•%Ö&ËÐM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112677250414272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114187774138368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1311966 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚749 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069210077184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 861133998080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 % _timestamp‚1.7845257208936973e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25930211811328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25069191464960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113531990824317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7368296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5979256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113332070033408 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7358141 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112677233374912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113332070033408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 _step‚59 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 855704104960 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 854399848667 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 854740410045 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25930344075264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1772656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114188132608000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5979254 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 856045535232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7368296 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5979254 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3084902 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113531633223579 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7358141 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1772936 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚ 685.161964106 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3084619 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 856045535232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113332087072768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896mv³Ú&ÌÔM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112677233374912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113332070033408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 _step‚59 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 855704104960 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 854399848667 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 854740410045 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25930344075264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1772656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114188132608000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5979254 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 856045535232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7368296 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5979254 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3084902 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113531633223579 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7358141 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1772936 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚ 685.161964106 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3084619 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 856045535232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113332087072768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112677250414272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114187774138368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1311966 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚749 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069210077184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 861133998080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 % _timestamp‚1.7845257208936973e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25930211811328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25069191464960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113531990824317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7368296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5979256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113332070033408 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7358141 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738;B!Ä0Í+ _wandbruntime‚685  _runtime‚685-Ô1Îú 2loss/SRqqt1/assignment_loss‚1.1262537220239537 1loss/SRqqt1/detection_loss‚0.7432463777345275 1loss/FBt2/assignment_loss‚0.41461009234664686 1loss/SRqqt2/detection_loss‚2.3190970604550576 /loss/FBt1/detection_loss‚3.4322195054513314 .loss/FBt2/detection_loss‚4.425752741957632 %loss/total_loss‚21.38653843890482 epoch‚1 % _timestamp‚1.7845257208950171e+09 /loss/FRt2/assignment_loss‚3.691705564098895 2loss/SRqqt2/assignment_loss‚1.3547754405338444 0loss/FBt1/assignment_loss‚0.1330991109792983 .loss/FRt2/detection_loss‚0.539821143020065 trainer/global_step‚749 _step‚60 0loss/FRt1/assignment_loss‚1.8738561669389704 /loss/FRt1/detection_loss‚1.3321015133646021 _runtime‚ 685.162789967l®Ïþ /loss/FRt1/detection_loss‚1.3321015133646021 _runtime‚ 685.162789967 2loss/SRqqt1/assignment_loss‚1.1262537220239537 1loss/SRqqt1/detection_loss‚0.7432463777345275 1loss/FBt2/assignment_loss‚0.41461009234664686 1loss/SRqqt2/detection_loss‚2.3190970604550576 /loss/FBt1/detection_loss‚3.4322195054513314 .loss/FBt2/detection_loss‚4.425752741957632 %loss/total_loss‚21.38653843890482 epoch‚1 % _timestamp‚1.7845257208950171e+09 /loss/FRt2/assignment_loss‚3.691705564098895 2loss/SRqqt2/assignment_loss‚1.3547754405338444 0loss/FBt1/assignment_loss‚0.1330991109792983 .loss/FRt2/detection_loss‚0.539821143020065 trainer/global_step‚749 _step‚60 0loss/FRt1/assignment_loss‚1.8738561669389704<ÁgÊd]Ђ(:Ò çöÒÜÍÊÞ disk.nvme1n1.in‚0) disk./.usagePercent‚76.19384407822423% disk./.usageGB‚1169.7659912109375 disk.dm-0.out‚ 3412.765625 disk.dm-1.in‚0& disk.nvme0c0n1.out‚2911.1181640625' disk.nvme1c1n1.out‚1040.64599609375% disk.nvme1n1.out‚1040.64599609375$ memory_percent‚99.97601509094238 disk.md0.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚0% proc.memory.availableMB‚ 3.9296875# proc.memory.rssMB‚ 2856.48046875) proc.memory.percent‚17.43457317352295 proc.cpu.threads‚17 network.recv‚226716 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 cpu‚25.689261310012583 network.sent‚ 1.056448e+06 disk.md0.out‚17.59375 disk.dm-0.in‚0! disk.dm-1.out‚501.07470703125 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚2911.1181640625 disk.rbd0.out‚0ÔY#Ò‘Ñ‚(:† çöÒ¦ÿß gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚289.515' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·8òo+Ò‚(:Î ¬çöÒæ—þÝ disk.rbd0.in‚0 disk.rbd1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 network.sent‚ 1.105718e+06) disk./.usagePercent‚76.19381078270665! disk.nvme0n1.out‚ 2912.1328125 disk.rbd0.out‚0 disk.rbd1.out‚0 disk.dm-0.out‚ 3413.328125% proc.memory.availableMB‚ 3.9140625# proc.memory.rssMB‚ 2856.48046875# disk.nvme0c0n1.out‚ 2912.1328125% disk.nvme1n1.out‚1043.00048828125 disk.md0.out‚ 18.06640625$ memory_percent‚99.97611045837402 cpu‚25.694381152030164 proc.cpu.threads‚17! disk.dm-1.out‚502.85986328125 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1043.00048828125 disk.nvme1n1.in‚0) proc.memory.percent‚17.43457317352295 network.recv‚230593$ disk./.usageGB‚1169.76548004150iE.4 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0þɨ‘Ó‚(:† ¬çöÒÁ¨ÏÞ gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚277.92' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.64 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çN†çbÔ‚(:× »çöÒœð³Þ disk.md0.in‚0 disk.dm-1.out‚ 502.982421875 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2914.30712890625% disk.nvme1c1n1.out‚1043.466796875 disk.nvme1n1.in‚0) disk./.usagePercent‚76.19381798845299 disk.md0.out‚ 18.20703125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0# disk.nvme1n1.out‚1043.466796875$ memory_percent‚99.97632503509521% disk./.usageGB‚1169.7655906677246 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 3.87890625# proc.memory.rssMB‚ 2856.48046875) proc.memory.percent‚17.43457317352295 cpu‚25.697860329037432 disk.dm-0.in‚0" disk.dm-0.out‚3415.52587890625' disk.nvme0c0n1.out‚2914.30712890625 proc.cpu.threads‚17 network.sent‚ 1.112398e+06 network.recv‚234058õkžÕ‚(:“ »çöÒîÕøÞ gpu.0.gpu‚35.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.703' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.56766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í8œ0Ö+ _wandbruntime‚728  _runtime‚728ðÂÎÆ×À % _timestamp‚1.7845257638841238e+09 _runtime‚ 728.152236184 _step‚61 ' lr-AdamW/pg1‚0.00017859765320580128 ' lr-AdamW/pg2‚0.00017859765320580128 trainer/global_step‚799hâQÍÊØÄ _runtime‚ 728.152236184 _step‚61 ' lr-AdamW/pg1‚0.00017859765320580128 ' lr-AdamW/pg2‚0.00017859765320580128 trainer/global_step‚799 % _timestamp‚1.7845257638841238e+09=î ¤0Ù+ _wandbruntime‚728  _runtime‚728AqüWÌ'ÚÆO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1396929 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26695043311104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6361113 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120810110754863 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7843278 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3281551 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27610731835392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7833137 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 910163107840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6361111 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105270272 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 909115235089 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14194248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1396932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121508042208768 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6361113 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120597537677824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 trainer/global_step‚799 _runtime‚ 728.153127146 _step‚62 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 915812406784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121507683746304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120810468355489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120597520638464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119901336081040 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121507683746304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123882496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1884907 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26695024698880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845257638851159e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120597520638464 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3281839 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7843278 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120597537677824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27610855717888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6361111 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 910504530944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7833137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 910504530944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121508042208768Ñí“Ð'ÛÊO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27610855717888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6361111 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 910504530944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7833137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 910504530944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121508042208768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1396929 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26695043311104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6361113 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120810110754863 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7843278 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3281551 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27610731835392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7833137 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 910163107840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6361111 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105270272 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 909115235089 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14194248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1396932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121508042208768 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6361113 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120597537677824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 trainer/global_step‚799 _runtime‚ 728.153127146 _step‚62 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 915812406784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121507683746304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120810468355489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120597520638464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119901336081040 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121507683746304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123882496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1884907 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26695024698880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845257638851159e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120597520638464 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3281839 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7843278 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120597537677824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472>­Îà³0Ü+ _wandbruntime‚728  _runtime‚728YyiÖ&ÝÐM [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27645154645504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26728223581184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1886967 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 916931064320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 trainer/global_step‚799 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7842793 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27645022381568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚63 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3285630 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120958649810587 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120745795140608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121657426817024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 728.769880143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26728204968960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120048766748512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 911273206784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1887262 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121657426817024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6368904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7845257645014513e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120048783787872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7852948 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 911273206784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6368906 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 916817412608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120745795140608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7852948 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14221854 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6368906 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528zþ}TÞÔM MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚636890 Ð?ņ&4 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7845257645014513e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120048783787872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7852948 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 911273206784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6368906 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 916817412608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120745795140608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7852948 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14221854 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6368906 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27645154645504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26728223581184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1886967 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 916931064320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 trainer/global_step‚799 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7842793 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27645022381568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚63 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3285630 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120958649810587 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120745795140608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121657426817024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 728.769880143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26728204968960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120048766748512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 911273206784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1887262 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121657426817024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697?Ók0ß+ _wandbruntime‚728  _runtime‚728-¢fÑþàø /loss/FRt2/assignment_loss‚3.742316508788643 /loss/FRt2/detection_loss‚0.5910698369748533 -loss/FBt2/detection_loss‚8.93588944911827 2loss/SRqqt2/assignment_loss‚1.3605959178045548 -loss/FRt1/detection_loss‚1.38692345233738 _step‚64 epoch‚1 2loss/SRqqt1/assignment_loss‚1.1361449872202192 1loss/FBt1/assignment_loss‚0.07918994053925592 0loss/FBt2/assignment_loss‚0.4882263219106657 1loss/SRqqt2/detection_loss‚1.8466116156652395 /loss/FBt1/detection_loss‚1.6322717229543975 trainer/global_step‚799 _runtime‚ 728.771496856 0loss/FRt1/assignment_loss‚1.9329965299105607 1loss/SRqqt1/detection_loss‚0.8809421730160693 %loss/total_loss‚24.01317845624011 % _timestamp‚1.7845257645025663e+09Kcÿtáü /loss/FRt2/detection_loss‚0.5910698369748533 -loss/FBt2/detection_loss‚8.93588944911827 2loss/SRqqt2/assignment_loss‚1.3605959178045548 -loss/FRt1/detection_loss‚1.38692345233738 _step‚64 epoch‚1 2loss/SRqqt1/assignment_loss‚1.1361449872202192 1loss/FBt1/assignment_loss‚0.07918994053925592 0loss/FBt2/assignment_loss‚0.4882263219106657 1loss/SRqqt2/detection_loss‚1.8466116156652395 /loss/FBt1/detection_loss‚1.6322717229543975 trainer/global_step‚799 _runtime‚ 728.771496856 0loss/FRt1/assignment_loss‚1.9329965299105607 1loss/SRqqt1/detection_loss‚0.8809421730160693 %loss/total_loss‚24.01317845624011 % _timestamp‚1.7845257645025663e+09 /loss/FRt2/assignment_loss‚3.742316508788643@¡QXaâ‚(:Ö ÊçöÒ¼Ñ’Þ disk.md0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2915.23291015625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚1043.998046875 network.sent‚ 1.115191e+06 network.recv‚236654) disk./.usagePercent‚76.19381674608293 disk.rbd1.in‚0# disk.nvme1n1.out‚1043.998046875 disk.md0.out‚ 18.56640625 disk.dm-1.in‚0$ memory_percent‚99.98457431793213 disk.rbd1.out‚0" disk.dm-0.out‚3416.13134765625 disk.dm-1.out‚ 503.060546875! proc.memory.rssMB‚ 2896.296875* proc.memory.percent‚17.677593231201172 cpu‚25.721544995547262 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2915.23291015625 disk.rbd0.in‚0& proc.memory.availableMB‚ 2.52734375% disk./.usageGB‚1169.7655715942383 disk.nvme1n1.in‚0uëÏgã‚(:’ ÊçöÒ¥ÚÇÞ gpu.0.gpu‚30.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.098' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.03266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é´G¶0ä+ _wandbruntime‚745  _runtime‚745—‹z³Ù=åÓ{ :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.30954164266586304 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3680555522441864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4518736004829407 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4296674132347107 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25397276878356934 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7626951336860657 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8999766111373901 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16512465476989746 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5983748435974121 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.524143636226654 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 )jet/accuracy_1_of_1‚0.243355393409729 *loss/val_total_loss‚21.365182156746677 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6552088856697083 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22740350663661957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19207724928855896 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9673739075660706 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9953740239143372 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01760270819067955 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40599218010902405 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1917429268360138 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5936129689216614 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3003993332386017 particle/accuracy_1_of_3‚1 _runtime‚ 745.898786693 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15649698674678802 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9141453504562378 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3198804259300232 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07817771285772324 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47684377431869507 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12455187737941742 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1732359677553177 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25699707865715027 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3996348977088928 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4927796721458435 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21693286299705505 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.967366635799408 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4149976372718811 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.39774495363235474 +jet/accuracy_2_of_3‚0.28571999073028564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6954541206359863 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.265625 +jet/accuracy_2_of_4‚0.31361156702041626 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_2‚0.6702824831008911 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.17684213817119598 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.43740373849868774 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9928958415985107 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.40241533517837524 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2332523912191391 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.033784329891204834 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.3609986901283264 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.021926768124103546 particle/accuracy_3_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 # _timestamp‚1.78452578163015e+09 _step‚65 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2582082152366638 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5validation_average_jet_accuracy‚0.280466228723526 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03440632298588753 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9925737977027893 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4922558069229126 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.651423990726471 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9859126210212708 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919893145561218 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4126887619495392 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08039236813783646 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.7668839693069458 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5186985731124878 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.3593103587627411 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028555484488606453 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3860357701778412 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4480747878551483 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24124376475811005 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3858333230018616 +jet/accuracy_3_of_4‚0.04001768305897713 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4388931691646576 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.39901870489120483 /particle/accuracy_3_of_3‚0.9980465769767761 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2246309071779251 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16452546417713165 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5887377262115479 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9638802409172058 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4368845522403717 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5177437663078308 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950203895568848 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3201828598976135 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9807997345924377 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,jet/accuracy_3_of_3‚0.039261527359485626 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9564697742462158 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20927712321281433 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16702087223529816 particle/accuracy_2_of_2‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.125 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 jet/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.46739277243614197 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9048510193824768 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.43983492255210876 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01833333447575569 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5132926106452942 trainer/global_step‚799 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2677343189716339 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9479497075080872 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3071562349796295 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5373403429985046 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17365975677967072 particle/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9720948338508606 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4759949743747711 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26034286618232727 -jet/accuracy_4_of_4‚0.0043290043249726295 epoch‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19976775348186493 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.977492094039917 +jet/accuracy_2_of_2‚0.08075591176748276 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9098919630050659 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4743868112564087 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4300113916397095 /particle/accuracy_4_of_4‚0.9473980665206909 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47603729367256165 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5545622706413269 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47394752502441406 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8097019195556641 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3732942044734955 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18960677087306976 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.3905518054962158 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6033635139465332 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23822394013404846 *jet/accuracy_1_of_3‚0.8090574145317078 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009205919690430164 *particle/specificity‚0.894537091255188 jet/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.390531450510025 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2677343189716339 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15866485238075256 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9936065673828125 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.524345874786377 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5285742878913879 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2883685529232025 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17125733196735382 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16512465476989746 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8215253949165344 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9914965629577637 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.1428571492433548 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.43740373849868774 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.30823788046836853 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.130204439163208 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 particle/accuracy_1_of_2‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007835268042981625 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01465968880802393 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2883685529232025 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4225383996963501 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43319520354270935 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9394179582595825 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8365635871887207 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2892952859401703 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9925737977027893 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.25231483578681946 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1976708173751831 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2301219254732132 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9924237132072449 5 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.98235684633255 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0833333358168602 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3079262375831604 &particle/f_score‚0.571645200252533 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4001987874507904 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.810706615447998Á("wæ×{ 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007835268042981625 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01465968880802393 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2883685529232025 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4225383996963501 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43319520354270935 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9394179582595825 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8365635871887207 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2892952859401703 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9925737977027893 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.25231483578681946 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1976708173751831 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2301219254732132 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9924237132072449 5 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.98235684633255 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0833333358168602 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3079262375831604 &particle/f_score‚0.571645200252533 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4001987874507904 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.810706615447998 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.30954164266586304 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3680555522441864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4518736004829407 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4296674132347107 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25397276878356934 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7626951336860657 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8999766111373901 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16512465476989746 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5983748435974121 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.524143636226654 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 )jet/accuracy_1_of_1‚0.243355393409729 *loss/val_total_loss‚21.365182156746677 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6552088856697083 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22740350663661957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19207724928855896 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9673739075660706 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9953740239143372 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01760270819067955 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40599218010902405 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1917429268360138 7 PuriÔ@óÖÙ-ty/*FRt*SRqqt2FBt/FRt_purity‚0.5936129689216614 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3003993332386017 particle/accuracy_1_of_3‚1 _runtime‚ 745.898786693 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15649698674678802 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9141453504562378 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3198804259300232 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07817771285772324 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47684377431869507 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12455187737941742 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1732359677553177 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25699707865715027 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3996348977088928 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4927796721458435 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21693286299705505 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.967366635799408 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4149976372718811 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.39774495363235474 +jet/accuracy_2_of_3‚0.28571999073028564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6954541206359863 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.265625 +jet/accuracy_2_of_4‚0.31361156702041626 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_2‚0.6702824831008911 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.17684213817119598 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.43740373849868774 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9928958415985107 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.40241533517837524 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2332523912191391 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.033784329891204834 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.3609986901283264 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.021926768124103546 particle/accuracy_3_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 # _timestamp‚1.78452578163015e+09 _step‚65 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2582082152366638 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5validation_average_jet_accuracy‚0.280466228723526 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03440632298588753 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9925737977027893 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4922558069229126 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.651423990726471 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9859126210212708 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919893145561218 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4126887619495392 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08039236813783646 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.7668839693069458 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5186985731124878 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.3593103587627411 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028555484488606453 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3860357701778412 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4480747878551483 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24124376475811005 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3858333230018616 +jet/accuracy_3_of_4‚0.04001768305897713 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4388931691646576 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.39901870489120483 /particle/accuracy_3_of_3‚0.9980465769767761 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2246309071779251 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16452546417713165 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5887377262115479 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9638802409172058 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4368845522403717 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5177437663078308 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950203895568848 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3201828598976135 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9807997345924377 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,jet/accuracy_3_of_3‚0.039261527359485626 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9564697742462158 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20927712321281433 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16702087223529816 particle/accuracy_2_of_2‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.125 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 jet/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.46739277243614197 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9048510193824768 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.43983492255210876 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01833333447575569 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5132926106452942 trainer/global_step‚799 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2677343189716339 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9479497075080872 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3071562349796295 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5373403429985046 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17365975677967072 particle/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9720948338508606 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4759949743747711 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26034286618232727 -jet/accuracy_4_of_4‚0.0043290043249726295 epoch‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19976775348186493 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.977492094039917 +jet/accuracy_2_of_2‚0.08075591176748276 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9098919630050659 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4743868112564087 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4300113916397095 /particle/accuracy_4_of_4‚0.9473980665206909 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47603729367256165 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5545622706413269 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47394752502441406 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8097019195556641 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3732942044734955 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18960677087306976 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.3905518054962158 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6033635139465332 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23822394013404846 *jet/accuracy_1_of_3‚0.8090574145317078 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009205919690430164 *particle/specificity‚0.894537091255188 jet/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.390531450510025 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2677343189716339 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15866485238075256 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9936065673828125 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.524345874786377 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5285742878913879 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2883685529232025 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17125733196735382 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16512465476989746 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8215253949165344 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9914965629577637 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.1428571492433548 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.43740373849868774 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.30823788046836853 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.130204439163208 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 particle/accuracy_1_of_2‚1Aöë«iç‚(:Þ ÙçöÒʱŽÞ disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1289.96337890625) disk./.usagePercent‚76.19381301897276% disk./.usageGB‚1169.7655143737793 disk.dm-1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0" proc.memory.rssMB‚ 2904.5703125' disk.nvme0c0n1.out‚2916.99462890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚1289.96337890625 disk.dm-0.in‚0" disk.dm-0.out‚3417.97119140625% disk.nvme0n1.out‚2916.99462890625 network.sent‚ 1.176586e+06 network.recv‚240949! disk.dm-1.out‚748.56494140625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0$ memory_percent‚96.98796272277832 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.md0.out‚ 18.73046875' proc.memory.availableMB‚ 493.4921875* proc.memory.percent‚17.728090286254883 cpu‚25.750158956786873”v!5“è‚(:ˆ ÙçöÒ˜é‡Þ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚268.614' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïp÷òdé‚(:Ù èçöÒ”Û±Þ! disk.dm-1.out‚748.60791015625$ memory_percent‚96.99850082397461 cpu‚25.7446978323869) disk./.usagePercent‚76.19381550371288$ disk./.usageGB‚1169.765552520752 disk.rbd0.in‚0' disk.nvme1c1n1.out‚1290.33154296875 disk.nvme1n1.in‚0 disk.dm-0.in‚0' disk.nvme0c0n1.out‚2917.72607421875& proc.memory.availableMB‚ 491.765625 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚3418.50244140625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2906.54296875 proc.cpu.threads‚17 network.sent‚ 1.233622e+06 disk.rbd1.out‚0 disk.md0.in‚0 disk.md0.out‚ 18.9609375 disk.nvme0c0n1.in‚0) proc.memory.percent‚17.74013042449951 network.recv‚245084% disk.nvme0n1.out‚2917.72607421875 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1290.33154296875uŠ,bžê‚(:“ èçöÒ¢¯©Þ gpu.0.gpu‚73.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.706' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.56866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶¡3òeë‚(:Ú ÷çöÒ¯ŠôÝ disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1536.1513671875$ memory_percent‚96.99270725250244* proc.memory.percent‚17.734313011169434 network.sent‚ 1.23854e+06 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0 cpu‚25.73855149156291% disk./.usageGB‚1169.7655754089355 disk.md0.in‚0 disk.nvme0c0n1.in‚0 disk.rbd1.out‚0# proc.memory.rssMB‚ 2905.58984375 proc.cpu.threads‚17 disk.md0.out‚ 19.0859375" disk.dm-0.out‚3420.21435546875 disk.dm-1.out‚994.1162109375 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2919.42138671875 disk.rbd0.in‚0$ disk.nvme1n1.out‚1536.1513671875( proc.memory.availableMB‚ 492.71484375 network.recv‚248691) disk./.usagePercent‚76.19381699455695 disk.dm-1.in‚0' disk.nvme0c0n1.out‚2919.42138671875r ý­“ì‚(:ˆ ÷çöÒ»ºÃÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.042' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Ž±õ0í+ _wandbruntime‚792  _runtime‚792u¶f.Åî¿ $ _timestamp‚1.784525827856002e+09 _runtime‚ 792.124518529 _step‚66 ' lr-AdamW/pg1‚0.00017854900376620302 ' lr-AdamW/pg2‚0.00017854900376620302 trainer/global_step‚849VzgÉïà ' lr-AdamW/pg1‚0.00017854900376620302 ' lr-AdamW/pg2‚0.00017854900376620302 trainer/global_step‚849 $ _timestamp‚1.784525827856002e+09 _runtime‚ 792.124518529 _step‚66B!PR>0ð+ _wandbruntime‚792  _runtime‚792+ÖŠ°Ë'ñÅO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8399283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784832 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992273982976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129194125462864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15272852 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130939639666688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2042496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130179044799619 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129953589621248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130939998129664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 986050045440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8409425 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8399283 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6873569 % _timestamp‚1.7845258278567684e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6873569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130939998129664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992377155584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6873571 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130939639666688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 986391469056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1487745 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚ 792.12491121 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1487748 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3529937 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 984902297395 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚67 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚849 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129953606660608 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6873571 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129194142502224 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 986050045440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29368890848256 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28376616865280 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 984561736125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129953606660608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304,&òÉO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129194142502224 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 986050045440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29368890848256 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28376616865280 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 984561736125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129953606660608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8399283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784832 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992273982976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129194125462864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15272852 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130939639666688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2042496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130179044799619 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129953589621248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130939998129664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 986050045440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8409425 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8399283 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6873569 % _timestamp‚1.7845258278567684e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6873569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130939998129664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992377155584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6873571 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130939639666688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 986391469056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMo•91©²nitor.on_train_batch_start/inactive_split.large_pool.freed‚1487745 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚ 792.12491121 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1487748 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3529937 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 984902297395 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚67 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚849 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129953606660608 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6873571 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272CŽMÙ…0ó+ _wandbruntime‚792  _runtime‚792¿LØòÕ&ôÏM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2044466 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993493603328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131089382737920 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6881364 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6881364 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8408939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130327226254713 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 987160144384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚68 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29403187573248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3533945 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 987160144384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130101864123392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15290301 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1489479 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6881362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29403309350912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130101881162752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 986010685759 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 985670124377 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2044764 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28409797135360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8419095 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28409815747584‡+UŒÙ&õÓM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2044764 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28409797135360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8419095 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28409815747584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2044466 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993493603328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131089382737920 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6881364 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6881364 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8408939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130327226254713 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 987160144384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚68 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29403187573248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3533945 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 987160144384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130101864123392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15290301 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1489479 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6881362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29403309350912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130101881162752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 986010685759 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 985670124377D>m5’0ö+ _wandbruntime‚792  _runtime‚792 "Â÷ú 0loss/FRt1/assignment_loss‚1.7410883152295673 0loss/FRt2/assignment_loss‚3.4281685943962175 1loss/FBt1/assignment_loss‚0.13618296046629663 0loss/FBt2/assignment_loss‚0.4501398269168533 2loss/SRqqt1/assignment_loss‚1.0487287971478914 2loss/SRqqt2/assignment_loss‚1.3107907825117482 /loss/FRt1/detection_loss‚1.3620779772600462 /loss/FRt2/detection_loss‚0.5933015960764338 1loss/SRqqt1/detection_loss‚0.8267522279264187 0loss/SRqqt2/detection_loss‚2.018220382233372 trainer/global_step‚849 epoch‚2 _runtime‚ 792.75941479 _step‚69 &loss/total_loss‚21.015585905775048 .loss/FBt1/detection_loss‚3.387064579960808 .loss/FBt2/detection_loss‚4.713069865649397 % _timestamp‚1.7845258284915998e+09(røþ &loss/total_loss‚21.015585905775048 .loss/FBt1/detection_loss‚3.387064579960808 .loss/FBt2/detection_loss‚4.713069865649397 % _timestamp‚1.7845258284915998e+09 0loss/FRt1/assignment_loss‚1.7410883152295673 0loss/FRt2/assignment_loss‚3.4281685943962175 1loss/FBt1/assignment_loss‚0.13618296046629663 0loss/FBt2/assignment_loss‚0.4501398269168533 2loss/SRqqt1/assignment_loss‚1.0487287971478914 2loss/SRqqt2/assignment_loss‚1.3107907825117482 /loss/FRt1/detection_loss‚1.3620779772600462 /loss/FRt2/detection_loss‚0.5933015960764338 1loss/SRqqt1/detection_loss‚0.8267522279264187 0loss/SRqqt2/detection_loss‚2.018220382233372 trainer/global_step‚849 epoch‚2 _runtime‚ 792.75941479 _step‚69E¶WŒdù‚(:Ù †èöÒŒ¾ÕÞ proc.cpu.threads‚17 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚994.1826171875 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 2920.42578125 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2905.58984375) disk./.usagePercent‚76.19381972777107 disk.md0.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 2920.42578125 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚96.99766635894775% disk./.usageGB‚1169.7656173706055 disk.md0.out‚ 19.53515625% disk.nvme1n1.out‚1536.77685546875 cpu‚25.733474475776163 network.sent‚ 1.257303e+06 network.recv‚251740" disk.dm-0.out‚3420.80810546875' disk.nvme1c1n1.out‚1536.77685546875 disk.rbd1.out‚0( proc.memory.availableMB‚ 491.90234375* proc.memory.percent‚17.734313011169434E.òžú‚(:“ †èöÒ‡á•ß gpu.0.gpu‚96.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.345' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.11500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Ap^û‚(:Ó •èöÒ§ýÞ" disk.nvme1n1.out‚ 1537.48046875 cpu‚25.72806287054325 proc.cpu.threads‚17 network.sent‚ 1.307571e+06% disk./.usageGB‚1169.7659606933594 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚2922.6552734375$ disk.nvme0n1.out‚2922.6552734375 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 1537.48046875) disk./.usagePercent‚76.19384209043214 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚0$ memory_percent‚96.99890613555908( proc.memory.availableMB‚ 491.69921875# proc.memory.rssMB‚ 2905.58203125* proc.memory.percent‚17.734265327453613 disk.dm-1.out‚ 994.337890625 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 19.7421875! disk.dm-0.out‚3423.1298828125 network.recv‚256076 disk.rbd0.in‚0 disk.rbd1.out‚0¬÷ižü‚(:“ •èöÒ¡½ÓÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚272.254' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.75133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dçôô]ý‚(:Ò ¤èöÒÔÃœÞ network.sent‚ 1.310493e+06 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.md0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚2924.1083984375* proc.memory.percent‚17.734313011169434 cpu‚25.723098250437555% disk./.usageGB‚1169.7668952941895 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.rbd1.out‚0! disk.nvme1n1.out‚ 1538.0859375 disk.dm-1.in‚0 disk.dm-1.out‚ 994.412109375 network.recv‚259305) disk./.usagePercent‚76.19390296656502$ disk.nvme0n1.out‚2924.1083984375# disk.nvme1c1n1.out‚ 1538.0859375 disk.nvme1n1.in‚0 disk.dm-0.in‚0$ memory_percent‚96.99902534484863' proc.memory.availableMB‚ 491.6796875 disk.md0.out‚ 20.13671875! disk.dm-0.out‚3424.2548828125# proc.memory.rssMB‚ 2905.58984375 proc.cpu.threads‚17ìñæòþ‚(:’ ¤èöÒ¦ÉùÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚119.758' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.919333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cÈ5¢0ÿ+ _wandbruntime‚837  _runtime‚837KoûXÆ€À _step‚70 ' lr-AdamW/pg1‚0.00017849462782318495 ' lr-AdamW/pg2‚0.00017849462782318495 trainer/global_step‚899 % _timestamp‚1.7845258733628836e+09 _runtime‚ 837.631018209–‚8ÉÊÄ ' lr-AdamW/pg1‚0.00017849462782318495 ' lr-AdamW/pg2‚0.00017849462782318495 trainer/global_step‚899 % _timestamp‚1.7845258733628836e+09 _runtime‚ 837.631018209 _step‚70F{ÂÐ0‚+ _wandbruntime‚837  _runtime‚837K^3Ö ƒËO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16147154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137367331767808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1041960570880 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3732019 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30035630369280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1040385510803 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1040044949533 _step‚71 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138409292338688 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1574448 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31083930960384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2157574 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1041619147264 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137367314728448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1041960570880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31083817564160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚899 _runtime‚ 837.63215372 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7263219 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8883935 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1048187194880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1041619147264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138408933875712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8894077 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3732308 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113396224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138409292338688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2157860 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30035648981504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176úç)Õ'„ÏO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2157574 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1041619147264 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137367314728448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1041960570880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31083817564160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚899 _runtime‚ 837.63215372 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7263219 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8883935 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1048187194880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1041619147264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138408933875712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8894077 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3732308 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113396224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138409292338688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2157860 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30035648981504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16147154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137367331767808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1041960570880 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3732019 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30035630369280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1040385510803 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1040044949533 _step‚71 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138409292338688 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1574448 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31083930960384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85G¤ÆüÆ0…+ _wandbruntime‚838  _runtime‚838¶æ Ü&†ÖM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16174761 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138558318476800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7271012 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2160160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1042729246208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2159886 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31118118734336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137515589230592 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3736065 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1042729246208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 838.263700378 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8903747 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1043070676992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30068829251584 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7271012 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137515606269952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1041493899167 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138558676946944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138558318476800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7271012 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2160160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1042729246208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2159886 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31118118734336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137515589230592 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3736065 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1042729246208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 838.263700378 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8903747 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1043070676992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30068829251584 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7271012 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137515606269952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1041493899167 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138558676946944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145878228084736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144781056875008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3934212 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1095528162941 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31694643873280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚75 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32798833046528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7652871 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1097529672704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1097188249088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143937209249424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144781056875008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9378729 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1104056902656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚ 883.201166307 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145878586547712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145878228084736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17031600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2273067 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚949 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32798700775936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9368587 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1104170561024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9368587 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144781039835648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1661145 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145878228084736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144781056875008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3934212 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1095528162941 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31694643873280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚75 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32798833046528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7652871 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1097529672704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1097188249088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143937209249424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144781056875008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9378729 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1104056902656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚ 883.201166307 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145878586547712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145878228084736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17031600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2273067 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚949 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32798700775936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9368587 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1104170561024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9368587 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144781039835648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1661145 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31727824143360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7660664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1105173025792 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146027612685824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytu s$ es.large_pool.allocated‚144084639916896 trainer/global_step‚949 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1096977112575 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144929331377152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2275365 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1098639778816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1098298348032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1098639778816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144084622877536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1098298348032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7660664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144929314337792 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 883.833969153 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146027971155968 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9378243 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31727842755584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3938244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1096636551193 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17038905 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9388399 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9378243 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1662879 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚76 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1662882 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32832997169152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7660662 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9388399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2275659 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145181617029471 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7660664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144929314337792 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 883.833969153 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146027971155968 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9378243 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31727842755584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3938244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1096636551193 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17038905 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9388399 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9378243 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1662879 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚76 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1662882 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32832997169152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7660662 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9388399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2275659 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145181617029471 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31727824143360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0LvN30š+ _wandbruntime‚883  _runtime‚883¸Û]þ›ø trainer/global_step‚949 $ _timestamp‚1.784525919566702e+09 _step‚77 /loss/FRt1/detection_loss‚1.3319591809699345 /loss/FBt2/detection_loss‚3.5382604472299692 0loss/FRt1/assignment_loss‚1.6029500333038535 /loss/FRt2/assignment_loss‚3.125985881421693 2loss/SRqqt1/assignment_loss‚0.9742457373233334 /loss/FRt2/detection_loss‚0.6051466025945251 %loss/total_loss‚18.73814984876741 2loss/SRqqt2/assignment_loss‚1.1911745394085205 1loss/FBt1/assignment_loss‚0.05216766052339268 0loss/FBt2/assignment_loss‚0.4484501368174928 /loss/SRqqt1/detection_loss‚0.85252069843583 .loss/FBt1/detection_loss‚3.381624112543818 epoch‚2 _runtime‚ 883.834563123 1loss/SRqqt2/detection_loss‚1.6336648181950475Ø#לü /loss/FRt2/detection_loss‚0.6051466025945251 %loss/total_loss‚18.73814984876741 2loss/SRqqt2/assignment_loss‚1.1911745394085205 1loss/FBt1/assignment_loss‚0.05216766052339268 0loss/FBt2/assignment_loss‚0.4484501368174928 /loss/SRqqt1/detection_loss‚0.85252069843583 .loss/FBt1/detection_loss‚3.381624112543818 epoch‚2 _runtime‚ 883.834563123 1loss/SRqqt2/detection_loss‚1.6336648181950475 trainer/global_step‚949 $ _timestamp‚1.784525919566702e+09 _step‚77 /loss/FRt1/detection_loss‚1.3319591809699345 /loss/FBt2/detection_loss‚3.5382604472299692 0loss/FRt1/assignment_loss‚1.6029500333038535 /loss/FRt2/assignment_loss‚3.125985881421693 2loss/SRqqt1/assignment_loss‚0.9742457373233334Mïú׆p‚(:å àèöÒ‰Ú¶Þ network.sent‚ 1.400523e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚0.328125 disk.rbd0.out‚0& proc.memory.availableMB‚ 491.515625" proc.memory.rssMB‚ 2905.5859375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚1550.392578125 disk.dm-0.in‚0.328125" disk.dm-0.out‚4271.47314453125 disk.dm-1.in‚0% disk.nvme0n1.out‚3771.39404296875$ memory_percent‚97.00002670288086 cpu‚25.70490690560319 network.recv‚273927( disk./.usagePercent‚76.2468530269462% disk./.usageGB‚1170.5798110961914 disk.rbd1.out‚0# disk.nvme1n1.out‚1550.392578125 disk.md0.in‚0 disk.dm-1.out‚995.2763671875' disk.nvme0c0n1.out‚3771.39404296875 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚0.328125 disk.rbd0.in‚0* proc.memory.percent‚17.734289169311523 proc.cpu.threads‚17 disk.md0.out‚25.75`ŽY›žž‚(:“ àèöÒ›Ï„ß gpu.0.gpu‚68.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚168.028' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚56.00933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê÷lQtŸ‚(:é ïèöÒ·’èÝ* proc.memory.percent‚17.734313011169434 proc.cpu.threads‚17) disk./.usagePercent‚76.24688706788581 disk.dm-0.in‚ 4.30859375 disk.dm-1.out‚995.7021484375& disk.nvme1c1n1.out‚1552.3212890625 disk.rbd1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 491.47265625 cpu‚25.700770500881156 network.recv‚278466% disk./.usageGB‚1170.5803337097168 disk.md0.out‚ 26.3515625 disk.dm-0.out‚4279.982421875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.30859375 disk.nvme0n1.out‚ 3779.65625# proc.memory.rssMB‚ 2905.58984375 disk.md0.in‚0 disk.nvme0c0n1.in‚ 4.30859375! disk.nvme0c0n1.out‚ 3779.65625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.rbd1.out‚0$ disk.nvme1n1.out‚1552.3212890625 network.sent‚ 1.449906e+06 disk.rbd0.in‚0$ memory_percent‚97.00028896331787VÊŸ ‚(:” ïèöÒœ°ÇÞ gpu.0.gpu‚82.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚134.676' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚44.891999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó\ŠÌ{¡‚(:ð þèöÒèç¤Þ disk.dm-1.in‚0 disk.dm-1.out‚995.7646484375 disk.nvme0c0n1.in‚ 4.30859375$ memory_percent‚97.00474739074707 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1553.1083984375 disk.rbd1.out‚0 disk.nvme1n1.in‚0* proc.memory.percent‚17.734265327453613 cpu‚25.69697265473812 disk.nvme0n1.in‚ 4.30859375 network.recv‚281686 disk.md0.in‚0 disk.dm-0.in‚ 4.30859375% disk.nvme0c0n1.out‚3788.099609375 disk.rbd0.in‚0 disk.rbd1.in‚0$ disk.nvme1n1.out‚1553.1083984375 network.sent‚ 1.455895e+06) disk./.usagePercent‚76.24691092139093% disk./.usageGB‚1170.5806999206543 disk.md0.out‚ 26.88671875# disk.nvme0n1.out‚3788.099609375 disk.rbd0.out‚0' proc.memory.availableMB‚ 490.7421875# proc.memory.rssMB‚ 2905.58203125 disk.dm-0.out‚4288.021484375ÿ®\D¢‚(:’ þèöÒ¸Š¶Þ gpu.0.gpu‚30.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.697' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.56566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?J´Ê0£+ _wandbruntime‚928  _runtime‚928U~µÆ¤À ' lr-AdamW/pg1‚0.00017836871076432843 ' lr-AdamW/pg2‚0.00017836871076432843 trainer/global_step‚999 % _timestamp‚1.7845259641433413e+09 _runtime‚ 928.411479974 _step‚78a±L—洀 ' lr-AdamW/pg1‚0.00017836871076432843 ' lr-AdamW/pg2‚0.00017836871076432843 trainer/global_step‚999 % _timestamp‚1.7845259641433413e+09 _runtime‚ 928.411479974 _step‚78NjXÝ0¦+ _wandbruntime‚928  _runtime‚928Ù9kh3!§ÌO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153347522293760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151308742623024 # _timestamp‚1.78452596414426e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8042521 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8042519 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33353675989504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152194781982208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153347522293760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚ 928.412405886 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152194764942848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33353657377280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1153098774528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4135735 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4135459 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8042519 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34513690414080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34513566532096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8042521ŒÎˆTŸ VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153347880756736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1152757350912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2387614 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1151011376349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9853239 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2387887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17895758 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153347880756736 Q4£Ö'¨ÐO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151308742623024 # _timestamp‚1.78452596414426e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8042521 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8042519 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33353675989504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152194781982208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153347522293760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚ 928.412405886 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152194764942848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33353657377280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1153098774528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4135735 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4135459 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8042519 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34513690414080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34513566532096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8042521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153347880756736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1152757350912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2387614 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1151011376349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9853239 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2387887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17895758 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153347880756736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153347522293760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999O‘»‰E0©+ _wandbruntime‚929  _runtime‚929&xéÃÞ&ªØM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17913207 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153497265364992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1154208880640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153497265364992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8050314 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2390168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152343056484352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1153867449856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152343039444992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151456173290496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152343039444992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1161130673664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161023313920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1152460325983 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152343056484352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17913207 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17923365 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9862895 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33386837647360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153496906894848 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4139467 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 _step‚80 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9873051 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4139750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34547986933248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8050312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152608633616479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1153867449856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1749579 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151456156251136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8050314 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4139467 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 _step‚80 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9873051 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4139750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34547986933248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8050312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152608633616479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1153867449856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1749579 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151456156251136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8050314 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152343056484352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1153867449856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152343039444992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151456173290496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152343039444992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1161130673664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161023313920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1152460325983 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152343056484352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17913207 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17923365 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9862895 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33386837647360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153496906894848P!›eR0¬+ _wandbruntime‚929  _runtime‚929äÈè*­ú /loss/FRt2/detection_loss‚0.6243888562056589 /loss/FBt1/detection_loss‚3.1497589017657166 0loss/FRt2/assignment_loss‚3.1354084704027865 2loss/SRqqt1/assignment_loss‚0.8448470010015536 1loss/FBt1/assignment_loss‚0.08890413982077294 .loss/FRt1/detection_loss‚1.403514119265085 .loss/FBt2/detection_loss‚5.301450857520104 _runtime‚ 929.048300417 _step‚81 0loss/FRt1/assignment_loss‚1.5589724338751068 1loss/FBt2/assignment_loss‚0.42061070669442413 1loss/SRqqt2/detection_loss‚1.3864066052764805 trainer/global_step‚999 1loss/SRqqt1/detection_loss‚0.8598978857803034 &loss/total_loss‚19.970246236402403 epoch‚2 " _timestamp‚1.7845259647801e+09 2loss/SRqqt2/assignment_loss‚1.1960862587944094ªÈD3®þ &loss/total_loss‚19.970246236402403 eè2rÑpoch‚2 " _timestamp‚1.7845259647801e+09 2loss/SRqqt2/assignment_loss‚1.1960862587944094 /loss/FRt2/detection_loss‚0.6243888562056589 /loss/FBt1/detection_loss‚3.1497589017657166 0loss/FRt2/assignment_loss‚3.1354084704027865 2loss/SRqqt1/assignment_loss‚0.8448470010015536 1loss/FBt1/assignment_loss‚0.08890413982077294 .loss/FRt1/detection_loss‚1.403514119265085 .loss/FBt2/detection_loss‚5.301450857520104 _runtime‚ 929.048300417 _step‚81 0loss/FRt1/assignment_loss‚1.5589724338751068 1loss/FBt2/assignment_loss‚0.42061070669442413 1loss/SRqqt2/detection_loss‚1.3864066052764805 trainer/global_step‚999 1loss/SRqqt1/detection_loss‚0.8598978857803034QXrý·ž¯‚(:“ éöÒÝ¥Õß gpu.0.gpu‚54.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.214' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.07133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚”…€°‚(:õ éöÒºíÚà$ memory_percent‚97.00589179992676 network.recv‚285044$ disk./.usageGB‚1170.579689025879 disk.rbd1.out‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 27.14453125" disk.dm-0.out‚4289.12353515625 disk.rbd0.in‚0& disk.nvme1c1n1.out‚1553.6279296875 disk.rbd1.in‚0 disk.dm-0.in‚ 4.30859375' disk.nvme0c0n1.out‚3789.38720703125% disk.nvme0n1.out‚3789.38720703125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 490.5546875" proc.memory.rssMB‚ 2905.5859375 network.sent‚ 1.459623e+06) disk./.usagePercent‚76.24684507577781 disk.dm-1.out‚995.9111328125 disk.nvme0c0n1.in‚ 4.30859375* proc.memory.percent‚17.734289169311523 cpu‚25.692657372912468 proc.cpu.threads‚17$ disk.nvme1n1.out‚1553.6279296875 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.30859375ÊËy‚±‚(:÷ œéöÒ§¤¡Þ proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme0n1.in‚ 4.30859375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚1554.2509765625" proc.memory.rssMB‚ 2905.5859375% disk.nvme0n1.out‚3790.29150390625 network.sent‚ 1.523881e+06 network.recv‚289020 disk.dm-0.in‚ 4.30859375" disk.dm-0.out‚4289.62744140625 disk.dm-1.out‚996.0478515625 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚76.24684035477159% disk./.usageGB‚1170.5796165466309 disk.md0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1554.2509765625 disk.rbd1.in‚0$ memory_percent‚98.51205348968506* proc.memory.percent‚17.734289169311523 disk.md0.out‚ 27.55078125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚3790.29150390625 disk.rbd1.out‚0( proc.memory.availableMB‚ 243.78515625 cpu‚25.688227121355897. 윲‚(:‘ œéöÒõ÷çÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.491' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.16366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Ü *~³‚(:ó «éöÒèàžÞ disk.nvme0c0n1.in‚ 4.30859375% disk.nvme0n1.out‚3792.48974609375$ disk.nvme1n1.out‚1554.8583984375" proc.memory.rssMB‚ 2905.5859375 network.sent‚ 1.528728e+06 network.recv‚292604 disk.md0.in‚0 disk.md0.out‚27.78125' disk.nvme0c0n1.out‚3792.48974609375 disk.nvme0n1.in‚ 4.30859375 disk.rbd0.out‚0( proc.memory.availableMB‚ 243.80859375" disk.dm-0.out‚4291.82177734375 disk.rbd0.in‚0# memory_percent‚98.5119104385376* proc.memory.percent‚17.734289169311523 proc.cpu.threads‚17) disk./.usagePercent‚76.24684457882978 disk.dm-1.out‚996.1435546875 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1554.8583984375 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚0 cpu‚25.683800097975137% disk./.usageGB‚1170.5796813964844 disk.dm-0.in‚ 4.30859375 disk.dm-1.in‚0oO[³ž´‚(:“ «éöÒð¿ÜÞ gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.209' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.06966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#¦tø0µ+ _wandbruntime‚974  _runtime‚974I >ƶÀ _runtime‚ 974.028572471 _step‚82 ' lr-AdamW/pg1‚0.00017829717773590738 ' lr-AdamW/pg2‚0.00017829717773590738 trainer/global_step‚1049 $ _timestamp‚1.784526009760445e+09L.ãÊ·Ä $ _timestamp‚1.784526009760445e+09 _runtime‚ 974.028572471 _step‚82 ' lr-AdamW/pg1‚0.00017829717773590738 ' lr-AdamW/pg2‚0.00017829717773590738 trainer/global_step‚1049R9Ѳ0¸+ _wandbruntime‚974  _runtime‚974khÙ'¹ÓO OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8432169 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2501682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚159887111147651 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18780204 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚83 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160817174965760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159608490050048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35012670881280 trainer/global_step‚1049 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1208326452736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160817174965760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10337891 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36228536471040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128076288 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8432171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10337891 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35012689493504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8432171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1208667876352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1208667876352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1834548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1834545 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10348033 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158680275996624 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8432169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159608490050048 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36228408394752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1215737513472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10348033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1206835151027 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159608507089408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159886753547021 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845260097614868e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4335946 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159608507089408 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160816816502784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4336230 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1215846977536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 974.029574753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206494589757 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1208326452736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109464064 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158680275996624 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8432169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159608490050048 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36228408394752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1215737513472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10348033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1206835151027 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159608507089408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159886753547021 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845260097614868e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4335946 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159608507089408 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160816816502784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4336230 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1215846977536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 974.029574753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206494589757 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1208326452736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109464064 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10347547 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8439964 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159756764552192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159756764552192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8439962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10357703 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4339938 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18787509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18797667 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8439964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160966201103872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4340220 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2503938 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1209777982464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10357703 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35045851151360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24i¢Á)K 566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1836282 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18787509 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159756781591552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069120 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36262700081664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1216958387200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1049 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158827689624736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1207943539391 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160966559574016 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 _runtime‚ 974.667744718qJ@Ðç&½áM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1209777982464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10357703 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35045851151360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1836282 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18787509 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159756781591552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069120 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36262700081664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1216958387200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1049 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158827689624736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1207943539391 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160966559574016 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 _runtime‚ 974.667744718 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10347547 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8439964 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159756764552192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159756764552192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8439962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10357703 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4339938 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18787509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18797667 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8439964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160966201103872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4340220 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2503938 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184T&ìÕ¾0¾+ _wandbruntime‚974  _runtime‚974 ½˜#¿ú 1loss/FBt1/assignment_loss‚0.08920664647624828 _step‚85 /loss/FRt2/assignment_loss‚3.087524506453129 2loss/SRqqt2/assignment_loss‚1.0520368187814573 /loss/FRt2/detection_loss‚0.6243090384321768 % _timestamp‚1.7845260104005654e+09 /loss/FRt1/assignment_loss‚1.534479566055726 0loss/FBt2/assignment_loss‚0.6471501102104135 1loss/SRqqt1/detection_loss‚0.8124844442327827 /loss/FBt1/detection_loss‚4.5103215677728254 &loss/total_loss‚20.058636878376515 epoch‚2 _runtime‚ 974.668425289 2loss/SRqqt1/assignment_loss‚0.9229520371253894 .loss/FRt1/detection_loss‚1.343607289090942 0loss/SRqqt2/detection_loss‚1.472525321079126 .loss/FBt2/detection_loss‚3.962039532666297 trainer/global_step‚1049~ ÏÀþ .loss/FBt2/detection_loss‚3.962039532666297 trainer/global_step‚1049 1loss/FBt1/assignment_loss‚0.08920664647624828 _step‚85 /loss/FRt2/assignment_loss‚3.087524506453129 2loss/SRqqt2/assignment_loss‚1.0520368187814573 /loss/FRt2/detection_loss‚0.6243090384321768 % _timestamp‚1.7845260104005654e+09 /loss/FRt1/assignment_loss‚1.534479566055726 0loss/FBt2/assignment_loss‚0.6471501102104135 1loss/SRqqt1/detection_loss‚0.8124844442327827 /loss/FBt1/detection_loss‚4.5103215677728254 &loss/total_loss‚20.058636878376515 epoch‚2 _runtime‚ 974.668425289 2loss/SRqqt1/assignment_loss‚0.9229520371253894 .loss/FRt1/detection_loss‚1.343607289090942 0loss/SRqqt2/detection_loss‚1.472525321079126U€Oký|Á‚(:ñ ºéöÒ¨­ñÝ# memory_percent‚98.5140323638916' proc.memory.availableMB‚ 243.4609375" proc.memory.rssMB‚ 2905.5859375 cpu‚25.679273457948938) disk./.usagePercent‚76.24685625710835 disk.nvme1n1.in‚0 disk.md0.in‚0* proc.memory.percent‚17.734289169311523 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0" disk.dm-0.out‚4292.37646484375 disk.dm-1.in‚0 disk.dm-1.out‚996.1982421875' disk.nvme0c0n1.out‚3793.31201171875 network.sent‚ 1.531692e+06 network.recv‚295986% disk./.usageGB‚1170.5798606872559 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1555.2587890625 disk.dm-0.in‚ 4.30859375 disk.nvme0c0n1.in‚ 4.30859375% disk.nvme0n1.out‚3793.31201171875$ disk.nvme1n1.out‚1555.2587890625 disk.md0.out‚28.0625 disk.nvme0n1.in‚ 4.30859375Q+ΤžÂ‚(:“ ºéöÒ…Ÿ¾Þ gpu.0.gpu‚75.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚259.684' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.56133333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Á]„Â(:ù ÉéöÒ­âµÞ disk.dm-0.in‚ 4.30859375! disk.dm-1.out‚996.32470703125* proc.memory.percent‚17.734289169311523) disk./.usagePercent‚76.24686917775696' disk.nvme1c1n1.out‚1556.63525390625 disk.md0.in‚0' disk.nvme0c0n1.out‚3797.22705078125' proc.memory.availableMB‚ 243.5703125% disk.nvme1n1.out‚1556.63525390625 network.sent‚ 1.596066e+06 network.recv‚300577% disk.nvme0n1.out‚3797.22705078125 disk.rbd0.in‚0" disk.dm-0.out‚4296.17431640625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.30859375% disk./.usageGB‚1170.5800590515137 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 28.71484375$ memory_percent‚98.51336479187012" proc.memory.rssMB‚ 2905.5859375 cpu‚25.675330375487928 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.30859375 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0ӿ۞Ă(:“ ÉéöÒÉÛ”ß gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.234' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.74466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lç'…uÅ‚(:ê ØéöÒ×´ƒÞ disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 1557.12109375 disk.md0.out‚ 29.0859375 disk.dm-0.in‚ 4.30859375 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 2905.5859375* proc.memory.percent‚17.734289169311523 cpu‚25.67179713039225) disk./.usagePercent‚76.24687191097108" disk.dm-0.out‚4296.77587890625 disk.rbd1.out‚0" disk.nvme1n1.out‚ 1557.12109375$ memory_percent‚98.51341247558594$ proc.memory.availableMB‚243.5625 network.sent‚ 1.598988e+06! disk.dm-1.out‚996.37158203125% disk.nvme0c0n1.out‚3798.173828125 disk.nvme0n1.in‚ 4.30859375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚303604% disk./.usageGB‚1170.5801010131836 disk.md0.in‚0 disk.nvme0c0n1.in‚ 4.30859375# disk.nvme0n1.out‚3798.173828125 disk.rbd0.out‚0Õµ*)žÆ‚(:“ ØéöÒ™ÉÈÞ gpu.0.gpu‚48.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.457' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.48566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r5œÃ2Ç- _wandbruntime‚1019  _runtime‚1019XqÞ†ÈÈ trainer/global_step‚1099 % _timestamp‚1.7845260550039992e+09 _runtime‚1019.272062326 _step‚86 ' lr-AdamW/pg1‚0.00017821993437838185 ' lr-AdamW/pg2‚0.00017821993437838185¾Ò­ÌÉÆ ' lr-AdamW/pg1‚0.00017821993437838185 ' lr-AdamW/pg2‚0.00017821993437838185 trainer/global_step‚1099 % _timestamp‚1.7845260550039992e+09 _runtime‚1019.272062326 _step‚86VðÁн2Ê- _wandbruntime‚1019  _runtime‚1019ââ9åÙ'ËÓO OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8821821 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1261977803165 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1019.276680864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167314127734659 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19644362 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168286110711808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _step‚87EÖÖnÌ×O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167022232196608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845260550047138e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168286110711808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _step‚87 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8821821 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1261977803165 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1019.276680864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167314127734659 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19644362 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 _runtime‚1019.907866933 _step‚88 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36704864655360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167170489659392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922982 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8829614 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166199240037696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37977603076608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1265347084288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1272841586688 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2618245 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1272738421248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4541224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167170506698752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36704883267584 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1265005653504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10842355 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166199222998336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167170506698752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1265347084288 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19661811 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8829612 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167462309189753 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168435853783040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4541486 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37977724854272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1263426752799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845260556395364e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167462666790495 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10832199 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8829612 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2618504 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19671969 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10832199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1265005653504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Äãæè&ÏâM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1265347084288 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19661811 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8829612 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167462309189753 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168435853783040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4541486 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37977724854272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1263426752799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845260556395364e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167462666790495 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10832199 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8829612 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2618504 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19671969 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10832199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1265005653504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 _runtime‚1019.907866933 _step‚88 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36704864655360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167170489659392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922982 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8829614 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166199240037696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37977603076608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1265347084288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1272841586688 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2618245 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1272738421248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4541224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167170506698752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36704883267584 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1265005653504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10842355 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166199222998336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167170506698752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Xô(IA2Ð- _wandbruntime‚1019  _runtime‚1019±Ÿt‹Ñý 0loss/FRt2/assignment_loss‚2.9833786700311946 1loss/SRqqt1/detection_loss‚0.8282371928071263 1loss/SRqqt2/detection_loss‚1.4026733329997787 .loss/FBt2/detection_loss‚1.817881730728217 _step‚89 0loss/FRt1/assignment_loss‚1.4672545973198636 .loss/FRt1/detection_loss‚1.374761143977235 &loss/total_loss‚14.185832441272655 epoch‚2 trainer/global_step‚1099 2loss/SRqqt1/assignment_loss‚0.7770415245982313 1loss/SRqqt2/assignment_loss‚1.061761909911882 1loss/FBt1/assignment_loss‚0.10929258763979588 /loss/FBt1/detection_loss‚1.3633000694497306 % _timestamp‚1.7845260556406686e+09 0loss/FBt2/assignment_loss‚0.4226855640633456 /loss/FRt2/detection_loss‚0.5775641177462553 _runtime‚1019.908482344MÎÒ /loss/FRt2/detection_loss‚0.5775641177462553 _runtime‚1019.908482344 0loss/FRt2/assignment_loss‚2.9833786700311946 1loss/SRqqt1/detection_loss‚0.8282371928071263 1loss/SRqqt2/detection_loss‚1.4026733329997787 .loss/FBt2/detection_loss‚1.817881730728217 _step‚89 0loss/FRt1/assignment_loss‚1.4672545973198636 .loss/FRt1/detection_loss‚1.374761143977235 &loss/total_loss‚14.185832441272655 epoch‚2 trainer/global_step‚1099 2loss/SRqqt1/assignment_loss‚0.7770415245982313 1loss/SRqqt2/assignment_loss‚1.061761909911882 1loss/FBt1/assignment_loss‚0.10929258763979588 /loss/FBt1/detection_loss‚1.3633000694497306 % _timestamp‚1.7845260556406686e+09 0loss/FBt2/assignment_loss‚0.4226855640633456YÝgBžÓ‚(:“ çéöÒ‡ç—Þ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.567' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.18900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0} @~Ô‚(:ó çéöÒÕ‡®ß disk.nvme1n1.in‚0% disk.nvme1n1.out‚1557.47705078125! disk.dm-1.out‚996.45849609375' disk.nvme1c1n1.out‚1557.47705078125 disk.rbd1.in‚0$ memory_percent‚98.51932525634766% proc.memory.availableMB‚ 242.59375# proc.memory.rssMB‚ 2906.54296875 disk.dm-0.in‚ 4.30859375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0) proc.memory.percent‚17.74013042449951 cpu‚25.668276418775587 proc.cpu.threads‚17$ disk.nvme0n1.out‚3799.0986328125 network.recv‚306459) disk./.usagePercent‚76.24687191097108% disk./.usageGB‚1170.5801010131836 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.30859375& disk.nvme0c0n1.out‚3799.0986328125 network.sent‚ 1.602927e+06 disk.md0.out‚ 29.23828125 disk.dm-0.out‚4297.634765625 disk.nvme0n1.in‚ 4.30859375 disk.rbd0.in‚0 disk.rbd1.out‚0;¼4~Õ‚(:ó öéöÒ¦µ¿ß disk.rbd0.out‚0$ memory_percent‚98.51808547973633 proc.cpu.threads‚17 network.recv‚310346) disk./.usagePercent‚76.24682246464275" disk.dm-1.out‚1028.53271484375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0% disk.nvme1n1.out‚1590.25439453125 disk.dm-0.in‚ 4.30859375 disk.nvme0c0n1.in‚ 4.30859375& disk.nvme0c0n1.out‚3801.2412109375 disk.rbd1.out‚0 disk.md0.in‚0& proc.memory.availableMB‚ 242.796875# proc.memory.rssMB‚ 2906.54296875 disk.dm-0.out‚ 4299.69921875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.30859375' disk.nvme1c1n1.out‚1590.25439453125 disk.nvme1n1.in‚0 disk.md0.out‚ 29.59765625) proc.memory.percent‚17.74013042449951 cpu‚25.66539302721577 network.sent‚ 1.667288e+06% disk./.usageGB‚1170.5793418884277$ disk.nvme0n1.out‚3801.2412109375i=rŸÖ‚(:” öéöÒþ‰à gpu.0.gpu‚38.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.531' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.843666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J~:>vׂ(:ë …êöÒÁôžÞ disk.dm-0.in‚ 4.30859375% disk./.usageGB‚1170.5794143676758' disk.nvme1c1n1.out‚1590.81298828125 disk.md0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2905.5859375 network.sent‚ 1.672144e+06) disk./.usagePercent‚76.24682718564897 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme0n1.in‚ 4.30859375$ memory_percent‚98.51226806640625" proc.memory.availableMB‚243.75 network.recv‚313868 disk.nvme1c1n1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1590.81298828125* proc.memory.percent‚17.734289169311523 cpu‚25.662426518571603 disk.rbd1.in‚0 disk.md0.out‚ 29.9453125 disk.dm-0.out‚ 4300.27734375" disk.dm-1.out‚1028.63330078125$ disk.nvme0c0n1.out‚ 3802.11328125" disk.nvme0n1.out‚ 3802.11328125 disk.rbd0.in‚0 proc.cpu.threads‚17³vŸžØ‚(:“ …êöÒ‡˜èÞ gpu.0.gpu‚51.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚278.047' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.68233333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹#vŒ=Ù‚(:ê ”êöÒåù£Þ network.recv‚316962! disk.dm-0.out‚4301.4931640625 disk.nvme0n1.in‚ 4.30859375 disk.rbd0.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 245.0390625 cpu‚25.664101716624547 proc.cpu.threads‚17% disk./.usageGB‚1170.5793952941895 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme0n1.out‚ 3803.671875 disk.rbd1.out‚0" proc.memory.rssMB‚ 2905.5859375 network.sent‚ 1.675066e+06 disk.dm-0.in‚ 4.30859375 disk.dm-1.in‚0" disk.nvme0c0n1.out‚ 3803.671875 disk.nvme1c1n1.in‚0# memory_percel¹8nt‚98.5044002532959* proc.memory.percent‚17.734289169311523( disk./.usagePercent‚76.2468259432789 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1591.39404296875 disk.md0.in‚0 disk.md0.out‚30.34375" disk.dm-1.out‚1028.70751953125 disk.rbd0.in‚0' disk.nvme1c1n1.out‚1591.39404296875nﱜÚ‚(:‘ ”êöÒç¦ùÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.726' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.90866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁSV2Û- _wandbruntime‚1065  _runtime‚1065ÙÜLÈÜ % _timestamp‚1.7845261007711198e+09 _runtime‚1065.039127537 _step‚90 ' lr-AdamW/pg1‚0.00017813698565294836 ' lr-AdamW/pg2‚0.00017813698565294836 trainer/global_step‚1149_r@ÌÝÆ _runtime‚1065.039127537 _step‚90 ' lr-AdamW/pg1‚0.00017813698565294836 ' lr-AdamW/pg2‚0.00017813698565294836 trainer/global_step‚1149 % _timestamp‚1.7845261007711198e+09Z]ʺ‰2Þ- _wandbruntime‚1065  _runtime‚1065茸ëÚ'ßÔO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175755763383808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319806080000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2731000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174435957303808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174741144321667 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 _runtime‚1065.039942758 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101075456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9211469 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175755763383808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9211471 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11317337 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9211469 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319806080000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39658163901952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9211471 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38330697889280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1317801577843 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1319464656384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39658283589632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20518664 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4738653 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173423325704464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11317337 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2730708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚91 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174435957303808 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20528808 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4738948 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38330716501504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2007945 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2007948 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11307195 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173423342743824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174435940264448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1317461016573 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚1149 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174435940264448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175755404920832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174740786721037 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1327567088128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175755404920832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1-Ó—Þ'àØO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101075456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9211469 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175755763383808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9211471 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11317337 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9211469 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319806080000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39658163901952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9211471 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38330697889280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1317801577843 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1319464656384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39658283589632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20518664 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4738653 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173423325704464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11317337 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2730708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚91 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174435957303808 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20528808 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4738948 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38330716501504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2007945 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2007948 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11307195 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173423342743824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174435940264448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1317461016573 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚1149 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174435940264448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175755404920832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174740786721037 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1327567088128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175755404920832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175755763383808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319806080000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2731000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174435957303808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174741144321667 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 _runtime‚1065.039942758 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184[~†@2á- _wandbruntime‚1065  _runtime‚1065÷-èÒã&âÝM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174889325776761 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173570773411296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11316851 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115486208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4742715 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39692576276992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11327007 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38363896771584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11316851 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174584214766592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2733036 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1328679505408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173570756371936 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9219264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174584231805952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9219264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20536113 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174584231805952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20536113 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39692460790784 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4742989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1328582631424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320574755328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175904789521920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320916186112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175905147992064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2009682 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38363878159360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚1149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11327007 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1320574755328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174889683377503 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1318909966207 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1318569404825”ÆãáM % _timestamp‚1.7845261014034839e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚1065.671740375 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2009679 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train— çžÍ"_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9219262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175905147992064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2009682 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38363878159360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚1149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11327007 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1320574755328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174889683377503 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1318909966207 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1318569404825 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174889325776761 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173570773411296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11316851 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115486208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4742715 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39692576276992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11327007 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38363896771584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11316851 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174584214766592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2733036 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1328679505408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173570756371936 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9219264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174584231805952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9219264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20536113 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174584231805952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20536113 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39692460790784 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4742989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1328582631424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320574755328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175904789521920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320916186112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EÚ2í- _wandbruntime‚1110  _runtime‚1110YüÊÈî ' lr-AdamW/pg1‚0.00017804833688724853 ' lr-AdamW/pg2‚0.00017804833688724853 trainer/global_step‚1199 % _timestamp‚1.7845261460631442e+09 _runtime‚1110.331325303 _step‚94ôÇAÌïÆ ' lr-AdamW/pg1‚0.00017804833688724853 ' lr-AdamW/pg2‚0.00017804833688724853 trainer/global_step‚1199 % _timestamp‚1.7845261460631442e+09 _runtime‚1110.331325303 _step‚94^èÍÍ<2ð- _wandbruntime‚1110  _runtime‚1110ñéÅÚ'ñÔO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180794876117424 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183224699129856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21403110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11801989 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845261460642903e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4940582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183225057592832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2094645 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2845652 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 trainer/global_step‚1199 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚1110.332505005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21392966 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1373284791251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181849682411008 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41373146480640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9601121 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41373020501504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1372944229981 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2094648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9601121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182168160908675 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39989730005504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2845934 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1383309108224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375033758208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39989711393280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181849665371648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183224699129856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181849665371648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9601119 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375375181824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375033758208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181849682411008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1383416475136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Cœ‹qòØO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181849665371648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183224699129856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181849665371648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9601119 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375375181824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375033758208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181849682411008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1383416475136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180794876117424 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183224699129856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21403110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11801989 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845261460642903e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4940582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183225057592832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2094645 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2845652 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 trainer/global_step‚1199 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚1110.332505005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21392966 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1373284791251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181849682411008 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41373146480640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9601121 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41373020501504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1372944229981 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2094648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.aëýšm ll.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9601121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182168160908675 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39989730005504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2845934 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1383309108224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375033758208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39989711393280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976_L±4d2ó- _wandbruntime‚1111  _runtime‚1111«~/ä&ôÞM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚1199 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4944255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2096382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41407309538304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40022910275584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117845504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1384417874944 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376485287936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183374083730944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180942289745536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376485287936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41407445996032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11811659 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182316699964511 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181997956913152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4944549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9608912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180942306784896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚1111.298747755 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1384535720448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181997956913152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2096379 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2848167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845261470304172e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚96 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21410415 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11801503 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9608914 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11811659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9608912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2847876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1374052618233 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180’Œf[è&õâM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1384417874944 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376485287936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183374083730944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180942289745536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376485287936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41407445996032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11811659 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182316699964511 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181997956913152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4944549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9608912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180942306784896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚1111.298747755 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1384535720448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181997956913152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2096379 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2848167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845261470304172e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚96 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21410415 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11801503 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9608914 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11811659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9608912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2847876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1374052618233 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚1199 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4944255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2096382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41407309538304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40022910275584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117845504`Ð(Ÿ»2ö- _wandbruntime‚1111  _runtime‚1111ô<@÷ÿ /loss/FRt2/detection_loss‚0.5287844086062082 1loss/SRqqt1/detection_loss‚0.7668216751809691 1loss/SRqqt2/detection_loss‚1.2615739235254588 /loss/FBt1/detection_loss‚2.2392490107175185 2loss/SRqqt1/assignment_loss‚0.7962034205683101 1loss/FBt1/assignment_loss‚0.12399719701411878 1loss/FBt2/assignment_loss‚0.19597986808152842 epoch‚2 trainer/global_step‚1199 /loss/FRt2/assignment_loss‚2.905697340693791 % _timestamp‚1.7845261470316567e+09 _step‚97 2loss/SRqqt2/assignment_loss‚0.8925507213767537 .loss/FRt1/detection_loss‚1.430747971017459 /loss/FBt2/detection_loss‚1.6011032115887194 &loss/total_loss‚14.240520782807533 _runtime‚1111.299580806 0loss/FRt1/assignment_loss‚1.4978120344366967³–Cý øƒ 2loss/SRqqt1/assignment_loss‚0.7962034205683101 1loss/FBt1/assignment_loss‚0.12399719701411878 1loss/FBt2/assignment_loss‚0.19597986808152842 epoch‚2 trainer/global_step‚1199 /loss/FRt2/assignment_loss‚2.905697340693791 % _timestamp‚1.7845261470316567e+09 _step‚97 2loss/SRqqt2/assignment_loss‚0.8925507213767537 .loss/FRt1/detection_loss‚1.430747971017459 /loss/FBt2/detection_loss‚1.6011032115887194 &loss/total_loss‚14.240520782807533 _runtime‚1111.299580806 0loss/FRt1/assignment_loss‚1.4978120344366967 /loss/FRt2/detection_loss‚0.5287844086062082 1loss/SRqqt1/detection_loss‚0.7668216751809691 1loss/SRqqt2/detection_loss‚1.2615739235254588 /loss/FBt1/detection_loss‚2.2392490107175185a‚$í|ù‚(:ñ ÐêöÒʪ·Þ disk.nvme0n1.in‚ 4.30859375$ proc.memory.availableMB‚242.1875" proc.memory.rssMB‚ 2908.2890625$ disk./.usageGB‚1169.767505645752 disk.nvme1n1.in‚0% disk.nvme0n1.out‚3819.09716796875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 34.67578125" disk.dm-0.out‚4314.95849609375! disk.dm-1.out‚1032.7822265625 disk.nvme0c0n1.in‚ 4.30859375' disk.nvme0c0n1.out‚3819.09716796875 cpu‚25.716140873750764 network.recv‚330623( disk./.usagePercent‚76.1939427224069 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1602.4052734375 disk.dm-0.in‚ 4.30859375 disk.rbd0.in‚0$ memory_percent‚98.52180480957031) proc.memory.percent‚17.75078773498535 network.sent‚ 1.764962e+06 disk.rbd1.in‚0 disk.rbd1.out‚0$ disk.nvme1n1.out‚1602.4052734375 disk.dm-1.in‚0,Бú‚(:’ ÐêöÒâðúÞ gpu.0.gpu‚33.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚281.386' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.79533333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"X€2û- _wandbruntime‚1128  _runtime‚1128#êé½üö{ ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9260151386260986 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4654195010662079 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2387995421886444 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17499801516532898 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11661342531442642 particle/accuracy_1_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.280509352684021 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2606666684150696 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07833333313465118 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.98247891664505 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9827573895454407 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.602024257183075 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6163090467453003 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6206339597702026 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5321816802024841 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4819326400756836 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9817428588867188 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9617505669593811 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6238598823547363 ) Purity/1FRt1SRqqt2FBt/FBt_purity‚0.75 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.633130669593811 ,jet/accuracy_3_of_3‚0.054270319640636444 +jet/accuracy_3_of_4‚0.06684397906064987 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1722380369901657 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9958426356315613 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8620749711990356 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00997862033545971 ,jet/accuracy_4_of_4‚0.002801120514050126 *loss/val_total_loss‚14.487132639761398 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14262154698371887 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9938904047012329 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4969954788684845 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03901151940226555 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.47291669249534607 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5594264268875122 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9526193737983704 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9853702783584595 +jet/accuracy_2_of_3‚0.48320016264915466 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961904883384705 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9437263011932373 jet/accuracy_4_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9608181715011597 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16959860920906067 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7696523070335388 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9923686981201172 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 $ _timestamp‚1.784526164467702e+09 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5601258277893066 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8719898462295532 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5749643445014954 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9340043663978577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5668371915817261 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.029191305860877037 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6389243006706238 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09286058694124222 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6145352721214294 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4347389340400696 particle/accuracy_2_of_4‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5979116559028625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5172766447067261 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7567375302314758 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8612490296363831 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6772956848144531 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (particle/accuracy‚0.8349260687828064 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.47710177302360535 +particle/sensitivity‚0.5953623652458191 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9924044013023376 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7255497574806213 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7870370745658875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_2_of_4‚0.7311684489250183 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.604210615158081 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18496881425380707 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 *jet/accuracy_1_of_2‚0.7415156960487366 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16097137331962585 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9957867860794067 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5650600790977478 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5052083730697632 particle/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt*FBt/F¸ÓlK?"Bt_purity‚0.983273446559906 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09286058694124222 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5831524729728699 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5142500400543213 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4233957529067993 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7884615659713745 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3611111044883728 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9875513911247253 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01908349059522152 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.12345665693283081 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5278071165084839 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2889319658279419 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23531249165534973 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.333516001701355 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6validation_average_jet_accuracy‚0.3324893116950989 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6193286180496216 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013175863772630692 *jet/accuracy_1_of_3‚0.8925736546516418 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12801112234592438 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9771339893341064 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5974760055541992 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1224190816283226 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23627808690071106 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8103452324867249 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6872472167015076 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17269732058048248 *jet/accuracy_1_of_1‚0.2893955707550049 particle/accuracy_1_of_4‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4307754635810852 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11460152268409729 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6747981309890747 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚1128.736437477 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1712971031665802 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1354166716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7947090864181519 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18743374943733215 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6111111044883728 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7266141772270203 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1657620221376419 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27240779995918274 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17667505145072937 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7076120376586914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 trainer/global_step‚1199 _step‚98 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5668371915817261 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950263500213623 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1861981898546219 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8132464289665222 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13058029115200043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6900755763053894 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.949472963809967 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6077078580856323 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7151563763618469 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.131026953458786 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5378566384315491 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9726651310920715 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7948790788650513 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5667534470558167 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1346631795167923 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17667505145072937 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17961251735687256 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7109804153442383 /particle/accuracy_3_of_3‚0.9990025758743286 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 epoch‚2 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9828528165817261 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9630076289176941 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6000000238418579 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3611111044883728 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9957867860794067 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16038841009140015 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5531855225563049 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_3_of_4‚0.9970238208770752 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9848256707191467 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11366729438304901 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.39055874943733215 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9851095080375671 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6359682083129883 *jet/accuracy_1_of_4‚0.9379453063011169 +particle/specificity‚0.9004470109939575 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12410987168550491 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006891815923154354 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.222619891166687 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.47515520453453064 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12558704614639282 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15644948184490204 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5590975880622864 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7510281801223755 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7510281801223755 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9722222089767456 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33762750029563904 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14541921019554138$5£Ò>ýú{ & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9340043663978577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5668371915817261 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.029191305860877037 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6389243006706238 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09286058694124222 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6145352721214294 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4347389340400696 particle/accuracy_2_of_4‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5979116559028625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5172766447067261 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7567375302314758 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8612490296363831 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6772956848144531 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (particle/accuracy‚0.8349260687828064 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.47710177302360535 +particle/sensitivity‚0.5953623652458191 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9924044013023376 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7255497574806213 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7870370745658875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_2_of_4‚0.7311684489250183 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.604210615158081 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18496881425380707 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 *jet/accuracy_1_of_2‚0.7415156960487366 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16097137331962585 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9957867860794067 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5650600790977478 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5052083730697632 particle/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.983273446559906 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09286058694124222 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5831524729728699 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5142500400543213 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4233957529067993 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7884615659713745 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3611111044883728 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9875513911247253 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01908349059522152 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.12345665693283081 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5278071165084839 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2889319658279419 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23531249165534973 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.333516001701355 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6validation_average_jet_accuracy‚0.3324893116950989 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6193286180496216 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013175863772630692 *jet/accuracy_1_of_3‚0.8925736546516418 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12801112234592438 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9771339893341064 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5974760055541992 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1224190816283226 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23627808690071106 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8103452324867249 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6872472167015076 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17269732058048248 *jet/accuracy_1_of_1‚0.2893955707550049 particle/accuracy_1_of_4‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4307754635810852 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11460152268409729 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6747981309890747 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚1128.736437477 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1712971031665802 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1354166716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7947090864181519 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18743374943733215 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6111111044883728 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7266141772270203 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1657620221376419 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27240779995918274 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17667505145072937 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7076120376586914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 trainer/global_step‚1199 _step‚98 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5668371915817261 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950263500213623 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1861981898546219 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8132464289665222 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13058029115200043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6900755763053894 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.949472963809967 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6077078580856323 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7151563763618469 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.131026953458786 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5378566384315491 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9726651310920715 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7948790788650513 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5667534470558167 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1346631795167923 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17667505145072937 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17961251735687256 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7109804153442383 /particle/accuracy_3_of_3‚0.9990025758743286 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 epoch‚2 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9828528165817261 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9630076289176941 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6000000238418579 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3611111044883728 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9957867860794067 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16038841009140015 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5531855225563049 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_3_of_4‚0.9970238208770752 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9848256707191467 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11366729438304901 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.39055874943733215 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9851095080375671 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6359682083129883 *jet/accuracy_1_of_4‚0.9379453063011169 +particle/specificity‚0.9004470109939575 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12410987168550491 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006891815923154354 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.222619891166687 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.47515520453453064 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12558704614639282 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15644948184490204 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5590975880622864 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7510281801223755 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7510281801223755 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9722222089767456 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33762750029563904 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14541921019554138 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9260151386260986 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4654195010662079 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2387995421886444 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17499801516532898 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11661342531442642 particle/accuracy_1_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.280509352684021 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2606666684150696 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07833333313465118 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.98247891664505 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9827573895454407 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.602024257183075 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6163090467453003 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6206339597702026 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5321816802024841 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4819326400756836 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9817428588867188 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9617505669593811 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6238598823547363 ) Purity/1FRt1SRqqt2FBt/FBt_purity‚0.75 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.633130669593811 ,jet/accuracy_3_of_3‚0.054270319640636444 +jet/accuracy_3_of_4‚0.06684397906064987 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1722380369901657 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9958426356315613 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8620749711990356 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00997862033545971 ,jet/accuracy_4_of_4‚0.002801120514050126 *loss/val_total_loss‚14.487132639761398 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14262154698371887 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9938904047012329 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4969954788684845 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03901151940226555 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.47291669249534607 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5594264268875122 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9526193737983704 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9853702783584595 +jet/accuracy_2_of_3‚0.48320016264915466 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961904883384705 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9437263011932373 jet/accuracy_4_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9608181715011597 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16959860920906067 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7696523070335388 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9923686981201172 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 $ _timestamp‚1.784526164467702e+09 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5601258277893066 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8719898462295532 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5749643445014954 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1bGyÁÜ€þ‚(:õ ßêöÒä©Þ disk.rbd1.out‚0$ memory_percent‚96.98514938354492& proc.memory.availableMB‚ 493.953125 cpu‚25.720476852324484 network.sent‚ 1.827509e+06% disk./.usageGB‚1169.7687530517578 disk.md0.in‚0 disk.dm-0.in‚ 4.30859375 disk.dm-1.out‚1278.259765625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚1848.4072265625# proc.memory.rssMB‚ 2898.98828125 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme1c1n1.out‚1848.4072265625) proc.memory.percent‚17.69402027130127 proc.cpu.threads‚17 network.recv‚334937 disk.md0.out‚ 34.94921875" disk.dm-0.out‚4316.61474609375 disk.nvme0c0n1.in‚ 4.30859375' disk.nvme0c0n1.out‚3820.84033203125% disk.nvme0n1.out‚3820.84033203125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0) disk./.usagePercent‚76.19402397340875 disk.nvme0n1.in‚ 4.30859375 disk.rbd1.in‚0ƒþÁ"žÿ‚(:“ ßêöÒÔøîÞ gpu.0.gpu‚97.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚273.384' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.12800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07 ‡H€‚(:ô îêöÒ˜ŒýÝ disk.rbd1.out‚0 disk.dm-0.in‚ 4.30859375 disk.dm-1.out‚1330.349609375 disk.nvme0c0n1.in‚ 4.30859375' disk.nvme0c0n1.out‚3823.07470703125 disk.md0.in‚0 disk.md0.out‚ 35.24609375" disk.dm-0.out‚4318.71728515625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.30859375 disk.rbd0.out‚0$ memory_percent‚96.99022769927979( proc.memory.availableMB‚ 493.12109375 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 1901.01171875 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 1901.01171875% disk.nvme0n1.out‚3823.07470703125 disk.rbd0.in‚0* proc.memory.percent‚17.699837684631348 network.recv‚338569% disk./.usageGB‚1169.7689170837402# proc.memory.rssMB‚ 2899.94140625 cpu‚25.717242496952508 proc.cpu.threads‚17 network.sent‚ 1.871791e+06) disk./.usagePercent‚76.19403465779125 disk.rbd1.in‚0žÍ‚(:’ îêöÒº‘ÊÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚118.979' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.659666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Ü…‚‚(:ú ýêöÒØõŠÞ disk.nvme1n1.in‚0 network.sent‚ 1.877775e+06 disk.md0.in‚0 disk.dm-0.in‚ 4.30859375 disk.nvme0c0n1.in‚ 4.30859375% disk.nvme1n1.out‚2629.78564453125' proc.memory.availableMB‚ 493.8671875# proc.memory.rssMB‚ 2898.94140625 cpu‚25.713437167412884) disk./.usagePercent‚76.19403888184945 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0* proc.memory.percent‚17.693734169006348 proc.cpu.threads‚17 disk.md0.out‚ 35.5859375' disk.nvme0c0n1.out‚3823.98193359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2629.78564453125$ memory_percent‚96.98567390441895 network.recv‚341911% disk./.usageGB‚1169.7689819335938" disk.dm-0.out‚4319.33447265625 disk.dm-1.in‚0" disk.dm-1.out‚2058.67724609375 disk.nvme0n1.in‚ 4.30859375% disk.nvme0n1.out‚3823.98193359375Í91×’ƒ‚(:‡ ýêöÒý¹ãÞ gpu.0.gpu‚25.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.562' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0# ï~2„- _wandbruntime‚1174  _runtime‚1174æß3ÁÆ…À _runtime‚1174.458536443 _step‚99 & lr-AdamW/pg1‚0.0001779539937750269 & lr-AdamW/pg2‚0.0001779539937750269 trainer/global_step‚1249 % _timestamp‚1.7845262101901588e+09Àƒ­Æ栀 _runtime‚1174.458536443 _step‚99 & lr-AdamW/pg1‚0.0001779539937750269 & lr-AdamW/pg2‚0.0001779539937750269 trainer/global_step‚1249 % _timestamp‚1.7845262101901588e+09cW~v42‡- _wandbruntime‚1174  _runtime‚11748÷õ!ˆØO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191205751393792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10113579 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191205751393792 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12357996 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1451262121472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41671322171904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41671303559680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191205734354432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1450920697344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10113577 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43131310226944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5186476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192657013515264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3001012 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191536379752174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5186185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.cIz½urrent‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚1249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1459988055040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚1174.460537469 _step‚100 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10113579 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191536737352808 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192656655051776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192657013515264 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22471573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784526210191075e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12357996 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3000724 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2185461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190087648459888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1448731292286 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43131186345472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191205734354432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1449071853560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10113577 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1459882785792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12368139 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344?‘â'‰ÜO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192657013515264 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22471573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784526210191075e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12357996 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3000724 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2185461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190087648459888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1448731292286 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43131186345472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191205734354432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1449071853560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10113577 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1459882785792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12368139 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191205751393792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10113579 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191205751393792 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12357996 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1451262121472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41671322171904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41671303559680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191205734354432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1450920697344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10113577 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43131310226944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5186476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192657013515264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3001012 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191536379752174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5186185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚1249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1459988055040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚1174.460537469 _step‚100 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10113579 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191536737352808 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192656655051776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496dZX2Š- _wandbruntime‚1175  _runtime‚11751D“vç&‹áM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22489022 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2187195 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192806398123520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22499181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1452372227584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12377809 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191354025895936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41704483829760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1452030796288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191685276408644 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191354008856576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10121372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10121370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43165604027904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1450180241924 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12367652 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1452030796288 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5190477 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41704502441984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5190187 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3003279 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43165494833664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 $ _timestamp‚1.784526210821029e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1175.089305729 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191684918807898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1449839680538 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191354025895936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚192806039652864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192806398123520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2187198 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12377809 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12367652 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10121372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190235096166720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452372227584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3002992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1461101585920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10121370 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10121370 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22499181 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1452372227584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12377809 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191354025895936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41704483829760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1452030796288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191685276408644 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191354008856576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10121372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10121370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43165604027904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 \JDeviceStatsMonitor.on_train_bÄB.‘Æatch_end/requested_bytes.small_pool.allocated‚ 1450180241924 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12367652 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1452030796288 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5190477 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41704502441984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5190187 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3003279 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43165494833664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 $ _timestamp‚1.784526210821029e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1175.089305729 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191684918807898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1449839680538 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191354025895936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚192806039652864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192806398123520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2187198 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0eþ@0B2- _wandbruntime‚1175  _runtime‚1175MÞûŽþ 1loss/FBt2/assignment_loss‚0.33870631322081735 /loss/FBt1/detection_loss‚1.3746509549024057 /loss/FBt2/detection_loss‚1.9328416125582797 epoch‚3 % _timestamp‚1.7845262108220797e+09 &loss/total_loss‚13.666482571949182 /loss/FRt2/assignment_loss‚2.853286999717235 2loss/SRqqt1/assignment_loss‚0.7012088233884424 2loss/SRqqt2/assignment_loss‚0.8954933471222485 0loss/FBt1/assignment_loss‚0.1377941448884218 /loss/FRt1/detection_loss‚1.3770565972569915 1loss/SRqqt1/detection_loss‚0.8383439409977707 0loss/SRqqt2/detection_loss‚1.259445184942352 .loss/FRt2/detection_loss‚0.613933658490228 trainer/global_step‚1249 _runtime‚1175.089871771 _step‚102 0loss/FRt1/assignment_loss‚1.3437209944639887Cq‚ 0loss/FBt1/assignment_loss‚0.1377941448884218 /loss/FRt1/detection_loss‚1.3770565972569915 1loss/SRqqt1/detection_loss‚0.8383439409977707 0loss/SRqqt2/detection_loss‚1.259445184942352 .loss/FRt2/detection_loss‚0.613933658490228 trainer/global_step‚1249 _runtime‚1175.089871771 _step‚102 0loss/FRt1/assignment_loss‚1.3437209944639887 1loss/FBt2/assignment_loss‚0.33870631322081735 /loss/FBt1/detection_loss‚1.3746509549024057 /loss/FBt2/detection_loss‚1.9328416125582797 epoch‚3 % _timestamp‚1.7845262108220797e+09 &loss/total_loss‚13.666482571949182 /loss/FRt2/assignment_loss‚2.853286999717235 2loss/SRqqt1/assignment_loss‚0.7012088233884424 2loss/SRqqt2/assignment_loss‚0.8954933471222485fÊâzšƒ‚(:ø ŒëöÒçÖüÝ disk.nvme0c0n1.in‚ 4.30859375 network.recv‚345467 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 36.0703125% disk.nvme0n1.out‚3828.66552734375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2631.03955078125( disk./.usagePercent‚76.1940398757455% disk./.usageGB‚1169.7689971923828' disk.nvme0c0n1.out‚3828.66552734375 disk.nvme0n1.in‚ 4.30859375 disk.rbd1.in‚0$ memory_percent‚96.98631763458252# proc.memory.rssMB‚ 2898.97265625 proc.cpu.threads‚17 disk.dm-0.in‚ 4.30859375" disk.dm-0.out‚4324.10009765625 disk.dm-1.in‚0" disk.dm-1.out‚2058.80224609375 disk.rbd1.out‚0( proc.memory.availableMB‚ 493.76171875 network.sent‚ 1.88268e+06% disk.nvme1n1.out‚2631.03955078125 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.69392490386963 cpu‚25.710153066167216¢æÁ‘‚(:’ ŒëöÒØÕŠß gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚272.54' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.84666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kgð8’‚(:ô ›ëöÒ⤰Þ network.recv‚349987) disk./.usagePercent‚76.19403863337544 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚2631.79345703125 disk.md0.in‚0" disk.dm-1.out‚2059.03271484375% disk./.usageGB‚1169.7689781188965 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme0n1.in‚ 4.30859375 disk.rbd1.in‚0% disk.nvme1n1.out‚2631.79345703125( proc.memory.availableMB‚ 494.01171875# proc.memory.rssMB‚ 2898.97265625 network.sent‚ 1.94583e+06& disk.nvme0c0n1.out‚3830.0576171875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 36.47265625 disk.dm-0.in‚ 4.30859375 disk.dm-0.out‚ 4325.1484375 disk.dm-1.in‚0$ disk.nvme0n1.out‚3830.0576171875 disk.rbd1.out‚0$ memory_percent‚96.98479175567627) proc.memory.percent‚17.69392490386963 cpu‚25.706926929708814 proc.cpu.threads‚17ôÍž“‚(:“ ›ëöÒøÔõÞ gpu.0.gpu‚77.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚288.755' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.25166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸5€”‚(:õ ªëöÒµ‹üÝ network.recv‚353281) disk./.usagePercent‚76.19407093499697 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.out‚ 36.79296875' disk.nvme0c0n1.out‚3832.72216796875$ disk./.usageGB‚1169.769474029541 disk.md0.in‚0 disk.dm-0.in‚ 4.30859375! disk.dm-0.out‚4327.7021484375" disk.dm-1.out‚2059.07568359375 disk.nvme0c0n1.in‚ 4.30859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.30859375% disk.nvme0n1.out‚3832.72216796875$ memory_percent‚96.98483943939209( proc.memory.availableMB‚ 494.00390625 proc.cpu.threads‚17% disk.nvme1c1n1.out‚2632.416015625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚2632.416015625# proc.memory.rssMB‚ 2898.97265625) proc.memory.percent‚17.69392490386963 cpu‚25.703409004557916 network.sent‚ 1.948954e+06¥¥‘“•‚(:ˆ ªëöÒíðÎÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚140.976' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚46.992 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pF(p2–- _wandbruntime‚1219  _runtime‚1219 ëjÇÉ—Ã % _timestamp‚1.7845262555416472e+09 _runtime‚1219.809762142 _step‚103 ' lr-AdamW/pg1‚0.00017785396237576535 ' lr-AdamW/pg2‚0.00017785396237576535 trainer/global_step‚1299µØÍ˜Ç ' lr-AdamW/pg1‚0.00017785396237576535 ' lr-AdamW/pg2‚0.00017785396237576535 trainer/global_step‚1299 % _timestamp‚1.7845262555416472e+09 _runtime‚1219.809762142 _step‚103g•ê¥Ë2™- _wandbruntime‚1219  _runtime‚1219ÓO+ß'šÙO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚200126307724288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5388069 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10503227 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198619459461632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200125949260800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2272164 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚1219.811838626 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10503229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44846231760896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197459181833488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10503227 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198619459461632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3115908 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200126307724288 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197459198872848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198619476500992 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5388366 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1515790815744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200125949260800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1506489799168 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44846107879424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3116202 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1504555066968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12842648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1506489799168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23345875 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43330335675904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198619476500992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10503229 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12852791 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1504214505694 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198963753939816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1515896084992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198963396339182 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12852791 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Š«o ã'›ÝO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1515896084992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198963396339182 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12852791 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚200126307724288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5388069 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10503227 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198619459461632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200125949260800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2272164 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚1219.811838626 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10503229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44846231760896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197459181833488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10503227 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198619459461632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3115908 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200126307724288 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197459198872848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198619476500992 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5388366 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1515790815744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200125949260800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1506489799168 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44846107879424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3116202 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1504555066968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12842648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1506489799168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23345875 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43330335675904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198619476500992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10503229 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12852791 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1504214505694 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198963753939816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0hy÷]Å2œ- _wandbruntime‚1220  _runtime‚1220£5ÚE@ãM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44880531792384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚1220.444768682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10511020 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚200275692332544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23363324 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12852304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43363515945984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚200275333861888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚105 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517015846400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23373483 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1505663455332 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5392075 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10511022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198767733963776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3118451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1505322893946 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚23fí&žçM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5392349 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3118180 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚105 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517015846400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23373483 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1505663455332 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5392075 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10511022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198767733963776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3118451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1505322893946 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44880531792384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚1220.444768682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10511020 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚200275692332544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23363324 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12852304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43363515945984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚200275333861888 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2358861 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1562058900992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5589685 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10892879 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206033201608192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46561112351232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206390770526824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204830715207088 % _timestamp‚1.7845263004466252e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207595243469824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207595601933312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24230322 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44989349179904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1562400325120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1559697719102 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206390412926190 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13337443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.frÁª×2Teed‚ 1571655804928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24220177 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13327300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10892879 &þ›ã'­ÝO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206033184568832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207595243469824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207595601933312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24230322 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44989349179904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1562400325120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1559697719102 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206390412926190 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13337443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1571655804928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24220177 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13327300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10892879 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13327300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1560038280376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2358861 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1562058900992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5589685 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10892879 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206033201608192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46561112351232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206390770526824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204830715207088 % _timestamp‚1.7845263004466252e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206181476110336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10900672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13347113 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1563510431232 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46595409431552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207744628070912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1572778913792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1560806107354 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24237626 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24247785 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3232740 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45022529449984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45022510837760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1572879981568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚109 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206181476110336 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5593335 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1561146668740 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1563168999936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10900672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1563168999936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _runtime‚1265.346054639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3233021 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207744986541568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224ïiOªí&°çM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5593619 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206539309582660 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1563510431232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚204978162913920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1349 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24237626 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2360595 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚109 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206181476110336 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5593335 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1561146668740 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1563168999936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10900672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1563168999936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _runtime‚1265.346054639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3233021 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207744986541568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207744986541568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206181476110336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10900672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13347113 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1563510431232 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46595409431552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207744628070912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1572778913792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1560806107354 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24237626 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24247785 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3232740 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45022529449984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45022510837760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1572879981568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215064537678848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46648362683904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215064896142336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213817787113832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213446909676032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1617628002816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5790716 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25104624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13822095 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11282527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213817429513198 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48275856808960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615521493784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213446926715392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1617969426944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11282527 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2445561 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46648344071680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212202265620048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184¥ŒÊqâ'¿ÜO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11282527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213817429513198 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48275856808960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615521493784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213446926715392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1617969426944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11282527 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2445561 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46648344071680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212202265620048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1627512737280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215064537678848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46648362683904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215064896142336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213817787113832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213446909676032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1617628002816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5790716 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25104624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13822095 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303puTCå2À- _wandbruntime‚1310  _runtime‚1310öÏ”‡¶ÁâM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213595184178176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213965968568922 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1619079533056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48310157163008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25111928 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616289320762 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 $ _timestamp‚1.784526346360854e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215214280750592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚113 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13821608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794735 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215213922279936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48310276843008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46681524341760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213966326169668 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212349679248160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11290320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215214280750592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213595201217536 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2447295 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11290322 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616289320762 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 $ _timestamp‚1.784526346360854e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215214280750592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚113 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13821608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794735 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215213922279936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48310276843008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46681524341760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213966326169668 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212349679248160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11290320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215214280750592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213595201217536 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2447295 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11290322 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1619079533056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48310157163008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25111928 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222533831887872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1673538528768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222534190351360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14306747 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1683320035328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2532264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48307357575680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221244446100206 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1670664145918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1673197104640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11672179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 % _timestamp‚1.7845263908642986e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49990805686784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222534190351360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49990677611008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48307376187904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11672179 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3459951 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1683429498880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2532261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5992212 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5992502 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219573781954288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221244803700840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25978926 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚1449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14306747 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚220860634783232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1673197104640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25968781 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220860634783232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚123740Ži}ÑÝO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25968781 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220860634783232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14296604 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222533831887872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14296604 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚1355.132611216 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222533831887872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1673538528768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222534190351360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14306747 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1683320035328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2532264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48307357575680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221244446100206 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1670664145918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1673197104640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11672179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 % _timestamp‚1.7845263908642986e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49990805686784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222534190351360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49990677611008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48307376187904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11672179 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3459951 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1683429498880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2532261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5992212 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5992502 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219573781954288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221244803700840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25978926 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚1449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14306747 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚220860634783232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1673197104640t@5H•2Ò- _wandbruntime‚1355  _runtime‚1355!+'²é&ÓãM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1672113095556 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1674307203584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14316417 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222683216488960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14316417 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14306260 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1674648634880 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5996207 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2533995 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3462212 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48340537845760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50024966283264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48340556457984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219721212621760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1671772534170 trainer/global_step‚1449 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11679972 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11679970 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222683574959616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2533998 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221392985155930 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221008926324736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684428437504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221393342756676 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 % _timestamp‚1.7845263914957523e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25996389 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221008909285376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221008926324736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221008909285376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1674307203584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219721229661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50025102740480 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25986230 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11679970 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684546282496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14306260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚117 _runtime‚1355.764031738 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1674648634880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5996500 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799‰¦Ýí&ÔçM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5996500 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1672113095556 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1674307203584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14316417 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222683216488960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14316417 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14306260 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1674648634880 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5996207 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2533995 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3462212 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48340537845760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50024966283264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48340556457984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219721212621760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1671772534170 trainer/global_step‚1449 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11679972 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11679970 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222683574959616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2533998 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221392985155930 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221008926324736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684428437504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221393342756676 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 % _timestamp‚1.7845263914957523e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25996389 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221008909285376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221008926324736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221008909285376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1674307203584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219721229661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50025102740480 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25986230 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11679970 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684546282496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14306260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚117 _runtime‚1355.764031738 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1674648634880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180u»ô"2Õ- _wandbruntime‚1355  _runtime‚1355uñÓÖþ % _timestamp‚1.7845263914968472e+09 0loss/SRqqt2/assignment_loss‚0.87246822532186 1loss/FBt1/assignment_loss‚0.11761400639364727 &loss/total_loss‚12.839243160135176 _step‚118 1loss/SRqqt2/detection_loss‚1.1604939687048976 0loss/FRt1/assignment_loss‚1.3904289740782518 1loss/FBt2/assignment_loss‚0.11536059502331601 /loss/FRt2/detection_loss‚0.5910840500993595 0loss/SRqqt1/detection_loss‚0.717938507426078 epoch‚3 _runtime‚1355.764650079 /loss/FRt2/assignment_loss‚2.738844345199497 2loss/SRqqt1/assignment_loss‚0.7095432977934463 /loss/FRt1/detection_loss‚1.3456490279836384 /loss/FBt1/detection_loss‚1.6192220687510706 /loss/FBt2/detection_loss‚1.4605960933601154 trainer/global_step‚1449 HØÿׂ 0loss/FRt1/assignment_loss‚1.3904289740782518 1loss/FBt2/assignment_loss‚0.11536059502331601 /loss/FRt2/detection_loss‚0.5910840500993595 0loss/SRqqt1/detection_loss‚0.717938507426078 epoch‚3 _runtime‚1355.764650079 /loss/FRt2/assignment_loss‚2.738844345199497 2loss/SRqqt1/assignment_loss‚0.7095432977934463 /loss/FRt1/detection_loss‚1.3456490279836384 /loss/FBt1/detection_loss‚1.6192220687510706 /loss/FBt2/detection_loss‚1.4605960933601154 trainer/global_step‚1449 % _timestamp‚1.7845263914968472e+09 0loss/SRqqt2/assignment_loss‚0.87246822532186 1loss/FBt1/assignment_loss‚0.11761400639364727 &loss/total_loss‚12.839243160135176 _step‚118 1loss/SRqqt2/detection_loss‚1.1604939687048976váFoä€Ø‚(:õ ÀìöÒÏδÞ network.recv‚388949% disk./.usageGB‚1169.5315284729004' disk.nvme1c1n1.out‚2638.33544921875% disk.nvme1n1.out‚2638.33544921875 disk.nvme0c0n1.in‚ 4.30859375# proc.memory.rssMB‚ 2899.94140625 disk.nvme0n1.in‚ 4.30859375 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚0! disk.dm-0.out‚4342.2490234375 disk.dm-1.in‚0% disk.nvme0c0n1.out‚3849.224609375 cpu‚25.67496957428082( disk./.usagePercent‚76.1785721200391# disk.nvme0n1.out‚3849.224609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.in‚0 disk.md0.out‚ 39.96484375 disk.dm-0.in‚ 4.30859375" disk.dm-1.out‚2060.00048828125 network.sent‚ 2.182366e+06 disk.nvme1c1n1.in‚0$ memory_percent‚98.49584102630615( proc.memory.availableMB‚ 246.44140625* proc.memory.percent‚17.699837684631348 proc.cpu.threads‚17b9,˜Ù‚(:’ ÀìöÒù¬øÞ gpu.0.gpu‚49.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.65' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.21666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Pq˃Ú‚(:ø ÏìöÒôÃóÝ cpu‚25.672214396405955% disk./.usageGB‚1169.5314712524414 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.30859375' proc.memory.availableMB‚ 246.4453125* proc.memory.percent‚17.699837684631348 proc.cpu.threads‚17$ disk.nvme0n1.out‚3850.0751953125' disk.nvme1c1n1.out‚2638.75341796875 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.30859375$ memory_percent‚98.49581718444824# proc.memory.rssMB‚ 2899.94140625 network.sent‚ 2.232175e+06) disk./.usagePercent‚76.17856839292894 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 40.25390625" disk.dm-1.out‚2060.04833984375 network.recv‚393558 disk.nvme0n1.in‚ 4.30859375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚2638.75341796875! disk.dm-0.out‚4342.8544921875 disk.dm-1.in‚0& disk.nvme0c0n1.out‚3850.0751953125õY“Û‚(:ˆ ÏìöÒ¥ßÅÞ gpu.0.gpu‚28.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.162' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ißi‚Ü‚(:÷ ÞìöÒòì‡Þ* proc.memory.percent‚17.699837684631348" disk.dm-0.out‚4414.09228515625 disk.nvme0c0n1.in‚ 4.30859375' disk.nvme0c0n1.out‚3865.65478515625 disk.nvme0n1.in‚ 4.30859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 cpu‚25.66984674269192 network.sent‚ 2.238253e+06 network.recv‚396920% disk./.usageGB‚1169.8335227966309 disk.dm-0.in‚ 4.30859375 disk.rbd1.in‚0' proc.memory.availableMB‚ 246.4453125 proc.cpu.threads‚17 disk.dm-1.in‚0% disk.nvme0n1.out‚3865.65478515625% disk.nvme1n1.out‚2695.31005859375$ memory_percent‚98.49581718444824# proc.memory.rssMB‚ 2899.94140625( disk./.usagePercent‚76.1982428136545" disk.dm-1.out‚2060.16943359375' disk.nvme1c1n1.out‚2695.31005859375 disk.rbd1.out‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 40.609375Ó6|2žÝ‚(:“ ÞìöÒÍ•àÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.508' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.16933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÝb%2Þ- _wandbruntime‚1399  _runtime‚1399jqcÉßà ' lr-AdamW/pg2‚0.00017739708787630734 trainer/global_step‚1499 % _timestamp‚1.7845264354694836e+09 _runtime‚1399.738558323 _step‚119 ' lr-AdamW/pg1‚0.00017739708787630734éäÁÍàÇ ' lr-AdamW/pg1‚0.00017739708787630734 ' lr-AdamW/pg2‚0.00017739708787630734 trainer/global_step‚1499 % _timestamp‚1.7845264354694836e+09 _runtime‚1399.738558323 _step‚119wó^ì2á- _wandbruntime‚1399  _runtime‚1399ñâØO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226945315327888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228274359890432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStat DP¿&sMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230003126096896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49966371079680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12061827 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1728766206464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228671462687214 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1726147359326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14791399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1499 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49966389691904 $ _timestamp‚1.784526435470287e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230003484560384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12061829 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12061827 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230003126096896 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6193071 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6193369 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3574110 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1729107630592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228671820287848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226945332367248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26843083 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6193071 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6193369 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3574110 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1729107630592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228671820287848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226945332367248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26843083 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230003126096896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49966371079680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12061827 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1728766206464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228671462687214 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1726147359326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180x[Æé¹2ä- _wandbruntime‚1400  _runtime‚1400Q/qRè&åâM trainer/global_step‚1499 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227092763034720 _runtime‚1400.369204909 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49999569961984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51739809029632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1730217736704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26860532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228422634392576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1727255747578 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1740257679872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _step‚121 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1729876305408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14801069 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228422634392576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230152869168640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6197374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3576676 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228422651431936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1729876305408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3576404 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49999551349760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227092745995360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845264361009188e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12069620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1730217736704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228422651431936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12069620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26860532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1740356650496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6197099 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2620698 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230152869168640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12069622 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12069622 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14790912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581h“S " ææM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1730217736704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228422651431936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12069620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26860532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1740356650496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6197099 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2620698 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230152869168640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12069622 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12069622 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14790912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 trainer/global_step‚1499 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227092763034720 _runtime‚1400.369204909 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49999569961984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51739809029632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1730217736704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26860532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228422634392576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1727255747578 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1740257679872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _step‚121 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1729876305408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14801069 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228422634392576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230152869168640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6197374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3576676 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228422651431936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1729876305408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3576404 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49999551349760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227092745995360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845264361009188e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12069620y/±pó2ç- _wandbruntime‚1400  _runtime‚1400Çj[¤èþ 2loss/SRqqt2/assignment_loss‚0.8327668609652285 /loss/FRt2/detection_loss‚0.6070382609090302 /loss/FBt1/detection_loss‚1.1215303214077617 _runtime‚ 1400.36980374 1loss/FBt1/assignment_loss‚0.10253168869344453 1loss/FBt2/assignment_loss‚0.22990704862210057 _step‚122 /loss/FRt1/detection_loss‚1.3850647731043455 1loss/SRqqt1/detection_loss‚0.7506573173704394 &loss/total_loss‚12.390206345433114 epoch‚3 trainer/global_step‚1499 % _timestamp‚1.7845264361020122e+09 /loss/FRt1/assignment_loss‚1.259100412608872 2loss/SRqqt1/assignment_loss‚0.7070197751961473 1loss/SRqqt2/detection_loss‚1.1538271028293148 /loss/FBt2/detection_loss‚1.5606912514742683 .loss/FRt2/assignment_loss‚2.68007153225216á•£é‚ .loss/FRt2/assignment_loss‚2.68007153225216 2loss/SRqqt2/assignment_loss‚0.8327668609652285 /loss/FRt2/detection_loss‚0.6070382609090302 /loss/FBt1/detection_loss‚1.1215303214077617 _runtime‚ 1400.36980374 1loss/FBt1/assignment_loss‚0.10253168869344453 1loss/FBt2/assignment_loss‚0.22990704862210057 _step‚122 /loss/FRt1/detection_loss‚1.3850647731043455 1loss/SRqqt1/detection_loss‚0.7506573173704394 &loss/total_loss‚12.390206345433114 epoch‚3 trainer/global_step‚1499 % _timestamp‚1.7845264361020122e+09 /loss/FRt1/assignment_loss‚1.259100412608872 2loss/SRqqt1/assignment_loss‚0.7070197751961473 1loss/SRqqt2/detection_loss‚1.1538271028293148 /loss/FBt2/detection_loss‚1.5606912514742683zç(õ܆ê‚(:û íìöÒä£þÝ network.sent‚ 2.243136e+06% disk./.usageGB‚1169.5963287353516 disk.nvme0n1.in‚ 4.30859375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚2695.99853515625% disk.nvme1n1.out‚2695.99853515625 disk.md0.out‚ 41.0546875 disk.dm-0.in‚ 4.30859375 network.recv‚400268) disk./.usagePercent‚76.18279294807697 disk.nvme0c0n1.in‚ 4.30859375' disk.nvme0c0n1.out‚3866.74853515625% disk.nvme0n1.out‚3866.74853515625 disk.nvme1c1n1.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚0" disk.dm-0.out‚4414.80712890625 disk.dm-1.in‚0" disk.dm-1.out‚2060.28369140625# proc.memory.rssMB‚ 2899.94140625* proc.memory.percent‚17.699837684631348 proc.cpu.threads‚17 disk.md0.in‚0$ memory_percent‚98.49622249603271( proc.memory.availableMB‚ 246.37890625 cpu‚25.667712820491595G'ªtë‚(:’ íìöÒºéõÞ gpu.0.gpu‚20.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.769' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.25633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Oá!Ctì‚(:é üìöÒ¿ç¢Þ( disk./.usagePercent‚76.1786098880889% disk./.usageGB‚1169.5321083068848 disk.md0.in‚0 disk.md0.out‚ 41.5546875 disk.dm-0.out‚4608.509765625 disk.nvme0c0n1.out‚ 4038.4375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2719.15283203125 network.recv‚404384% disk.nvme1n1.out‚2719.15283203125" disk.dm-1.out‚2060.35791015625 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.49627017974854* proc.memory.percent‚17.699837684631348 proc.cpu.threads‚17 disk.rbd1.out‚0 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme0n1.in‚ 4.30859375( proc.memory.availableMB‚ 246.37109375# proc.memory.rssMB‚ 2899.94140625 cpu‚25.66560990750167 network.sent‚ 2.304531e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 4.30859375 disk.dm-1.in‚0 disk.nvme0n1.out‚ 4038.4375&ùí…ží‚(:“ üìöÒð—óÞ gpu.0.gpu‚68.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚253.523' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.50766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02°ƒî‚(:ø ‹íöÒ¤›Þ network.recv‚407836) disk./.usagePercent‚76.17860640945273! disk.dm-1.out‚2060.4521484375 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme0n1.in‚ 4.30859375% disk.nvme0n1.out‚4039.19873046875' disk.nvme1c1n1.out‚2719.60595703125$ memory_percent‚98.49627017974854$ disk./.usageGB‚1169.532054901123% disk.nvme1n1.out‚2719.60595703125# proc.memory.rssMB‚ 2899.94140625* proc.memory.percent‚17.699837684631348 cpu‚25.663227227916245 proc.cpu.threads‚17 network.sent‚ 2.311351e+06 disk.md0.in‚0 disk.md0.out‚ 41.81640625 disk.dm-0.out‚4609.076171875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚4039.19873046875 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.30859375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 246.37109375> Øø“ï‚(:ˆ ‹íöÒ†€ÉÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚180.798' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚60.266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VB°N2ð- _wandbruntime‚1444  _runtime‚14449p«ÆñÀ & lr-AdamW/pg1‚0.0001772687187063197 & lr-AdamW/pg2‚0.0001772687187063197 trainer/global_step‚1549 % _timestamp‚1.7845264804760654e+09 _runtime‚ 1444.74432682 _step‚123ô¢™ÊòÄ & lr-AdamW/pg1‚0.0001772687187063197 & lr-AdamW/pg2‚0.0001772687187063197 trainer/global_step‚1549 % _timestamp‚1.7845264804760654e+09 _runtime‚ 1444.74432682 _step‚123{"5)2ó- _wandbruntime‚1444  _runtime‚1444Lù¹ïÞ'ôØO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2705664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3688892 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237472420305920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15276051 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234316848701488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚1549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12451477 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51625384583680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236098479274222 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784676732416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1784335308288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 _runtime‚1444.745450272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235688102036992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237472778769408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12451477 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1781630572734 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6394556 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6394279 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688618 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237472420305920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12451479 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15276051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1784676732416 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845264804773257e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2705661 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12451479 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235688084997632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚236098836874856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51625403195904ÖC–ák#õÜO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27717385 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚237472778769408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235688102036992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15265908 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53420474057216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234316865740848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1784335308288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845264804773257e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2705661 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12451479 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235688084997632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚236098836874856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51625403195904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2705664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3688892 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237472420305920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15276051 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234316848701488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚1549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12451477 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51625384583680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236098479274222 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784676732416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1784335308288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 _runtime‚1444.745450272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235688102036992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237472778769408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12451477 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1781630572734 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6394556 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6394279 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688618 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237472420305920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12451479 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15276051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1784676732416 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12459270 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691148 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51658564853760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27734834 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6398262 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2707395 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15275564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2707398 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1796084234240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27744993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237622163377664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15275564 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3690867 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53454649088000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845264811091022e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1785786838528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237622163377664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1785786838528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51658583465984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1796185302016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1783079522372 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1549 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15285721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚1445.377451555 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12459272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12459270 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12459270 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691148 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51658564853760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27734834 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6398262 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2707395 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15275564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2707398 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1796084234240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27744993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237622163377664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15275564 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3690867 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53454649088000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845264811091022e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1785786838528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237622163377664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1785786838528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51658583465984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1796185302016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1783079522372 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1549 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15285721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚1445.377451555 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12459272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12459270 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3}‹ or2ù- _wandbruntime‚1445  _runtime‚1445:Ћóú€ 2loss/SRqqt1/assignment_loss‚0.6913889106393072 2loss/SRqqt2/assignment_loss‚0.8770404771110455 1loss/FBt1/assignment_loss‚0.11778923578919534 /loss/FRt2/detection_loss‚0.5923401710030071 &loss/total_loss‚12.626696580608355 1loss/FBt2/assignment_loss‚0.15443476988246654 /loss/FBt1/detection_loss‚1.1257886704941253 epoch‚3 /loss/FRt2/assignment_loss‚2.750588100304365 .loss/FRt1/detection_loss‚1.336132889090591 1loss/SRqqt1/detection_loss‚0.8094985307795468 1loss/SRqqt2/detection_loss‚1.0983373866709742 /loss/FBt2/detection_loss‚1.8970400019019926 trainer/global_step‚1549 _runtime‚1445.378110076 0loss/FRt1/assignment_loss‚1.1763174369417388 % _timestamp‚1.7845264811102781e+09 _step‚126ÍV û„ 1loss/FBt2/assignment_loss‚0.15443476988246654 /loss/FBt1/detection_loss‚1.1257886704941253 epoch‚3 /loss/FRt2/assignment_loss‚2.750588100304365 .loss/FRt1/detection_loss‚1.336132889090591 1loss/SRqqt1/detection_loss‚0.8094985307795468 1loss/SRqqt2/detection_loss‚1.0983373866709742 /loss/FBt2/detection_loss‚1.8970400019019926 trainer/global_step‚1549 _runtime‚1445.378110076 0loss/FRt1/assignment_loss‚1.1763174369417388 % _timestamp‚1.7845264811102781e+09 _step‚126 2loss/SRqqt1/assignment_loss‚0.6913889106393072 2loss/SRqqt2/assignment_loss‚0.8770404771110455 1loss/FBt1/assignment_loss‚0.11778923578919534 /loss/FRt2/detection_loss‚0.5923401710030071 &loss/total_loss‚12.626696580608355~ «°ü‚(:ô šíöÒßÅ‘Þ network.recv‚411258 disk.md0.in‚0" disk.dm-0.out‚4627.68017578125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme1n1.out‚2720.6416015625! disk.dm-1.out‚2060.5263671875' disk.nvme0c0n1.out‚4057.24072265625 disk.rbd1.in‚0 disk.rbd1.out‚0# memory_percent‚98.4905481338501) proc.memory.percent‚17.69390106201172% disk.nvme0n1.out‚4057.24072265625) disk./.usagePercent‚76.17864889850874% disk./.usageGB‚1169.5327072143555 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 proc.cpu.threads‚17 disk.md0.out‚ 41.98828125 disk.dm-0.in‚ 4.30859375 disk.nvme0n1.in‚ 4.30859375& disk.nvme1c1n1.out‚2720.6416015625( proc.memory.availableMB‚ 247.30859375 proc.memory.rssMB‚ 2898.96875 cpu‚25.661048847932726 network.sent‚ 2.314273e+06ŠYÈÏœý‚(:‘ šíöÒÊ´Žß gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.724' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.24133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oQ€Žƒþ‚(:ø ©íöÒÕˆ°Þ disk.nvme0c0n1.in‚ 4.30859375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 42.19921875" disk.dm-0.out‚4628.90283203125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚4058.55126953125$ memory_percent‚98.49059581756592( proc.memory.availableMB‚ 247.30078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.rbd1.out‚0 disk.nvme0n1.in‚ 4.30859375# proc.memory.rssMB‚ 2898.97265625 cpu‚25.658910331132482 proc.cpu.threads‚17 network.recv‚415378) disk./.usagePercent‚76.17866778253364& disk.nvme1c1n1.out‚2721.0849609375 disk.rbd1.in‚0 disk.dm-0.in‚ 4.30859375% disk.nvme0n1.out‚4058.55126953125) proc.memory.percent‚17.69392490386963 network.sent‚ 2.378584e+06% disk./.usageGB‚1169.5329971313477$ disk.nvme1n1.out‚2721.0849609375! disk.dm-1.out‚2060.5771484375ß~f€ÿ‚(:‚ ©íöÒÑÞóÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.5' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.5 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KñiÚ{€ ‚(:ð ¸íöÒê°“Þ disk.md0.out‚ 42.484375! disk.dm-1.out‚2060.6201171875 disk.nvme0n1.in‚ 4.30859375 disk.rbd1.in‚0$ memory_percent‚98.49061965942383$ disk.nvme1n1.out‚2722.2392578125! disk.dm-0.out‚4641.2001953125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.30859375& disk.nvme1c1n1.out‚2722.2392578125# proc.memory.rssMB‚ 2898.97265625) proc.memory.percent‚17.69392490386963 proc.cpu.threads‚17 network.sent‚ 2.381506e+06 disk.dm-0.in‚ 4.30859375& disk.nvme0c0n1.out‚4070.3427734375 disk.nvme1c1n1.in‚0 disk.rbd1.out‚0 cpu‚25.65699762766164 network.recv‚418818 disk.md0.in‚0$ disk.nvme0n1.out‚4070.3427734375 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 247.296875) disk./.usagePercent‚76.19607860501203% disk./.usageGB‚1169.8002967834473 disk.nvme1n1.in‚0tbþž ‚(:“ ¸íöÒ“ªåÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.715' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.23833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í×C˜2‚ - _wandbruntime‚1489  _runtime‚1489GS½¬Çƒ Á & lr-AdamW/pg1‚0.0001771347052632292 & lr-AdamW/pg2‚0.0001771347052632292 trainer/global_step‚1599 % _timestamp‚1.7845265251293824e+09 _runtime‚1489.397531633 _step‚127…øLË„ Å & lr-AdamW/pg2‚0.0001771347052632292 trainer/global_step‚1599 % _timestamp‚1.7845265251293824e+09 _runtime‚1489.397531633 _step‚127 & lr-AdamW/pg1‚0.00017713470526322926)‚2… - _wandbruntime‚1489  _runtime‚1489<#2‡† ØO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15750560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243525853461864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1839904410112 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55135307575296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1837113786142 % _timestamp‚1.7845265251304822e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15760703 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12841127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6595602 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53284416699904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12841127 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243101810104832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244941714514944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12841129 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1599 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1850793994752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1837454347416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2792364 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1850890875392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28601832 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6595312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53284398087680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243525495861230 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12841129 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647FÉ0ºã'‡ ÝO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244941714514944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12841129 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1599 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1850793994752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1837454347416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2792364 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1850890875392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28601832 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6595312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53284398087680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243525495861230 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12841129 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15750560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243525853461864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1839904410112 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55135307575296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1837113786142 % _timestamp‚1.7845265251304822e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15760703 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12841127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6595602 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53284416699904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12841127 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243101810104832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245091457586688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3805205 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1841014509056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1841355940352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243250084606976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2794098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241835829781920 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15770373 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12848922 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55169605617152 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6599568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53317596969984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243674392517700 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1852008647168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28609136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12848920 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚129 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243250084606976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92679168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚ 1490.02914721 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243250101646336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1841355940352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6599300 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55169494325760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241835812742560 trainer/global_step‚1599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243250101646336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15770373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243674034916954 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12848922 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245091457586688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15760216Ñ@j芊 æM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12848922 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245091457586688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15760216 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245091457586688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3805205 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1841014509056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1841355940352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243250084606976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2794098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241835829781920 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15770373 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12848922 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55169605617152 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6599568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53317596969984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243674392517700 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1852008647168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28609136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12848920 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚129 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243250084606976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92679168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚ 1490.02914721 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243250101646336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1841355940352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6599300 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55169494325760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241835812742560 trainer/global_step‚1599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243250101646336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15770373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243674034916954Or]±2‹ - _wandbruntime‚1490  _runtime‚1490¸¸”Œ ü 0loss/FRt1/assignment_loss‚1.1112563970013403 2loss/SRqqt2/assignment_loss‚0.7428970810568255 /loss/FRt1/detection_loss‚1.3295294401284248 -loss/FBt1/detection_loss‚2.38284969064501 &loss/total_loss‚12.792765888652632 epoch‚3 trainer/global_step‚1599 2loss/SRqqt1/assignment_loss‚0.6215857954684505 0loss/FBt2/assignment_loss‚0.4336273539329984 1loss/SRqqt2/detection_loss‚1.0442079317641324 $ _timestamp‚1.784526525761981e+09 _runtime‚1490.029828431 /loss/FRt2/assignment_loss‚2.743080248630125 0loss/FBt1/assignment_loss‚0.0974959464972813 /loss/FRt2/detection_loss‚0.5707351521106715 1loss/SRqqt1/detection_loss‚0.7143744308899591 /loss/FBt2/detection_loss‚1.0011264205274146 _step‚130^iX®  /loss/FRt2/assignment_loss‚2.743080248630125 0loss/FBt1/assignment_loss‚0.0974959464972813 /loss/FRt2/detection_loss‚0.5707351521106715 1loss/SRqqt1/detection_loss‚0.7143744308899591 /loss/FBt2/detection_loss‚1.0011264205274146 _step‚130 0loss/FRt1/assignment_loss‚1.1112563970013403 2loss/SRqqt2/assignment_loss‚0.7428970810568255 /loss/FRt1/detection_loss‚1.3295294401284248 -loss/FBt1/detection_loss‚2.38284969064501 &loss/total_loss‚12.792765888652632 epoch‚3 trainer/global_step‚1599 2loss/SRqqt1/assignment_loss‚0.6215857954684505 0loss/FBt2/assignment_loss‚0.4336273539329984 1loss/SRqqt2/detection_loss‚1.0442079317641324 $ _timestamp‚1.784526525761981e+09 _runtime‚1490.029828431‚ø=Š‚Ž ‚(:÷ ÇíöÒɼÎÞ disk.rbd0.in‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚2786.82861328125# memory_percent‚98.5008955001831( proc.memory.availableMB‚ 245.61328125" proc.memory.rssMB‚ 2900.3359375 network.recv‚422151 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.30859375* proc.memory.percent‚17.702245712280273 cpu‚25.671090689213973% disk./.usageGB‚1169.7709846496582 disk.md0.out‚ 43.06640625 disk.dm-0.in‚ 4.30859375! disk.dm-1.out‚2060.8076171875 disk.nvme0c0n1.in‚ 4.30859375 disk.nvme1n1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.400351e+06) disk./.usagePercent‚76.19416933070563 disk.dm-0.out‚4888.283203125' disk.nvme0c0n1.out‚4254.25732421875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2786.82861328125 disk.rbd1.in‚0 disk.md0.in‚0% disk.nvme0n1.out‚4254.25732421875¼›ý´“ ‚(:ˆ ÇíöÒŸ¼‹ß gpu.0.gpu‚77.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.628' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.876 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>¸¼À2 - _wandbruntime‚1507  _runtime‚1507  ;>‘ þ{ & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7626523375511169 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6042543053627014 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8267385363578796 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9874999523162842 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5364583730697632 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7981858849525452 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9721567034721375 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17047765851020813 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7082602381706238 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0344553105533123 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1793910712003708 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.506276547908783 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9918020367622375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9756823778152466 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1592053771018982 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20834732055664062 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03794294595718384 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7989681959152222 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.133055180311203 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08898557722568512 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6196932792663574 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8153141736984253 ,jet/accuracy_3_of_3‚0.060003507882356644 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5656545758247375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9929416179656982 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937974810600281 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027652844786643982 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9745863676071167 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6282015442848206 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2011350840330124 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06013903021812439 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6323228478431702 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8552480340003967 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9847861528396606 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9948300123214722 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07887408137321472 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7093926668167114 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20953264832496643 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1509145051240921 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9924079179763794 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3687790334224701 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48297008872032166 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _runtime‚ 1507.10040788 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872408509254456 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_5‚1 epoch‚3 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6631485819816589 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7029180526733398 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1623736172914505 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6202419400215149 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8153141736984253 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6754439473152161 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9907776713371277 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19966758787631989 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 /particle/accuracy_3_of_4‚0.9936224818229675 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9923686981201172 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7733509540557861 +particle/sensitivity‚0.6393634080886841 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9929416179656982 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17758488655090332 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9871731996536255 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02526412159204483 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3564814627170563 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8167611360549927 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17750386893749237 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16736899316310883 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5579365491867065 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7157042026519775 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6282015442848206 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19966758787631989 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.984502911567688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9908287525177002 jet/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15608039498329163 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7139933705329895 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_4‚0.07830311357975006 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29900357127189636 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6681798696517944 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.539353609085083 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5061955451965332 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 *loss/val_total_loss‚13.418195713440715 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6235328912734985 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5155518054962158 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6267531514167786 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06013903021812439 +particle/specificity‚0.8952926397323608 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6613530516624451 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1491086333990097 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08874257653951645 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8934310674667358 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19844575226306915 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0788494125008583 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8455914855003357 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5583975315093994 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6244678497314453 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2691830098628998 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7132249474525452 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9887681007385254 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_1_of_1‚0.31432345509529114 particle/accuracy_2_of_5‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18481911718845367 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.384348601102829 *jet/accuracy_2_of_3‚0.5163408517837524 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6661685109138489 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6698639988899231 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8659123182296753 /particle/accuracy_2_of_2‚0.9998962879180908 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47744059562683105 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19084107875823975 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9937613606452942 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7636553645133972 +jet/accuracy_2_of_2‚0.13242606818675995 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1893094778060913 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2640250027179718 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5872233510017395 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017551248893141747 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37360262870788574 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1956087052822113 /particle/accuracy_3_of_3‚0.9990025758743286 % _timestamp‚1.7845265428318174e+09 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13959285616874695 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4024386703968048 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_4‚0.9707254767417908 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6028324961662292 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚1599 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9818496704101562 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4937500059604645 *jet/accuracy_1_of_3‚0.9125361442565918 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.210344135761261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 4validation_average_jet_accuracy‚0.35664102435112 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6487136483192444 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.609886646270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5411966443061829 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36786144971847534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1772603839635849 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7089848518371582 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7107031941413879 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8134921193122864 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9780547618865967 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4074956774711609 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26025620102882385 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19210004806518555 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6096938848495483 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 /particle/accuracy_4_of_4‚0.9717927575111389 (particle/accuracy‚0.8403321504592896 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012915692292153835 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7148265838623047 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9887095093727112 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5097993016242981 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6040346622467041 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5366240739822388 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9671978950500488 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.949172854423523 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841044545173645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_2‚0.7784623503684998 _step‚131 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9968254566192627¿X‹Ù-’ ƒ| 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2691830098628998 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7132249474525452 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9887681007385254 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_1_of_1‚0.31432345509529114 particle/accuracy_2_of_5‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18481911718845367 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.384348601102829 *jet/accuracy_2_of_3‚0.5163408517837524 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6661685109138489 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6698639988899231 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8659123182296753 /particle/accuracy_2_of_2‚0.9998962879180908 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47744059562683105 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19084107875823975 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9937613606452942 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7636553645133972 +jet/accuracy_2_of_2‚0.13242606818675995 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1893094778060913 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2640250027179718 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5872233510017395 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017551248893141747 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37360262870788574 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1956087052822113 /particle/accuracy_3_of_3‚0.9990025758743286 % _timestamp‚1.7845265428318174e+09 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13959285616874695 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4024386703968048 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_4‚0.9707254767417908 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6028324961662292 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚1599 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9818496704101562 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4937500059604645 *jet/accuracy_1_of_3‚0.9125361442565918 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.210344135761261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 4validation_average_jet_accuracy‚0.35664102435112 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6487136483192444 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.609886646270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5411966443061829 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36786144971847534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1772603839635849 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7089848518371582 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7107031941413879 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8134921193122864 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9780547618865967 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4074956774711609 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26025620102882385 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19210004806518555 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6096938848495483 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 /particle/accuracy_4_of_4‚0.9717927575111389 (particle/accuracy‚0.8403321504592896 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012915692292153835 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7148265838623047 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9887095093727112 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5097993016242981 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6040346622467041 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5366240739822388 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9671978950500488 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.949172854423523 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841044545173645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_2‚0.7784623503684998 _step‚131 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9968254566192627 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7626523375511169 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6042543053627014 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8267385363578796 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9874999523162842 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5364583730697632 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7981858849525452 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9721567034721375 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17047765851020813 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7082602381706238 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0344553105533123 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1793910712003708 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.506276547908783 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9918020367622375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9756823778152466 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1592053771018982 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20834732055664062 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03794294595718384 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7989681959152222 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.133055180311203 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08898557722568512 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6196932792663574 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8153141736984253 ,jet/accuracy_3_of_3‚0.060003507882356644 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5656545758247375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9929416179656982 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937974810600281 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027652844786643982 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9745863676071167 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6282015442848206 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2011350840330124 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06013903021812439 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6323228478431702 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8552480340003967 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9847861528396606 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9948300123214722 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07887408137321472 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7093926668167114 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20953264832496643 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1509145051240921 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9924079179763794 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3687790334224701 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48297008872032166 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _runtime‚ 1507.10040788 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872408509254456 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_5‚1 epoch‚3 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6631485819816589 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7029180526733398 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1623736172914505 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6202419400215149 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8153141736984253 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6754439473152161 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9907776713371277 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19966758787631989 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.76851850¼zÓP074806213 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 /particle/accuracy_3_of_4‚0.9936224818229675 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9923686981201172 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7733509540557861 +particle/sensitivity‚0.6393634080886841 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9929416179656982 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17758488655090332 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9871731996536255 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02526412159204483 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3564814627170563 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8167611360549927 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17750386893749237 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16736899316310883 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5579365491867065 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7157042026519775 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6282015442848206 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19966758787631989 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.984502911567688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9908287525177002 jet/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15608039498329163 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7139933705329895 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_4‚0.07830311357975006 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29900357127189636 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6681798696517944 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.539353609085083 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5061955451965332 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 *loss/val_total_loss‚13.418195713440715 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6235328912734985 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5155518054962158 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6267531514167786 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06013903021812439 +particle/specificity‚0.8952926397323608 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6613530516624451 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1491086333990097 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08874257653951645 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8934310674667358 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19844575226306915 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0788494125008583 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8455914855003357 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5583975315093994 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6244678497314453ƒ[A»”“ ‚(:‰ ÖíöÒþ’÷Ý' disk.nvme1c1n1.out‚3039.34716796875 disk.rbd1.in‚0 disk.nvme1n1.in‚ 0.09765625 disk.dm-0.in‚ 4.5703125! disk.dm-0.out‚5016.7529296875& disk.nvme0c0n1.out‚4380.2978515625$ disk.nvme0n1.out‚4380.2978515625) disk./.usagePercent‚76.24706323596014 disk.md0.in‚0" disk.dm-1.out‚2306.45068359375 disk.nvme0c0n1.in‚ 4.47265625 disk.nvme0n1.in‚ 4.47265625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2901.26171875 cpu‚25.678744245098823 disk.md0.out‚ 45.24609375 disk.rbd0.out‚0$ memory_percent‚95.48404216766357 proc.cpu.threads‚17 network.sent‚ 2.448792e+06% disk./.usageGB‚1170.5830383300781 disk.nvme1c1n1.in‚ 0.09765625 disk.rbd1.out‚0% disk.nvme1n1.out‚3039.34716796875 disk.dm-1.in‚0( proc.memory.availableMB‚ 739.89453125) proc.memory.percent‚17.70789623260498 network.recv‚426415@á­Ÿ” ‚(:” ÖíöÒó´íÞ gpu.0.gpu‚87.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.629' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.876333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œqª<‚• ‚(:÷ åíöÒ—íÞ disk.dm-0.in‚ 7.36328125% disk.nvme1c1n1.out‚3285.888671875# disk.nvme1n1.out‚3285.888671875( proc.memory.availableMB‚ 740.83203125" disk.nvme0n1.out‚ 5101.77734375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚ 0.19921875$ memory_percent‚95.47832012176514* proc.memory.percent‚17.702078819274902 cpu‚25.67684173454372 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚45.71875 disk.nvme0c0n1.in‚ 7.1640625$ disk.nvme0c0n1.out‚ 5101.77734375 disk.nvme0n1.in‚ 7.1640625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 0.19921875 disk.rbd1.out‚0 network.sent‚ 2.50796e+06 network.recv‚430640( disk./.usagePercent‚76.2470833623551 disk.dm-0.out‚5737.931640625 disk.dm-1.in‚0 disk.dm-1.out‚ 2552.30859375# proc.memory.rssMB‚ 2900.30859375% disk./.usageGB‚1170.5833473205566åø(– ‚(:’ åíöÒÈü×Þ gpu.0.gpu‚90.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚275.84' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.94666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q¡¬6ƒ— ‚(:ø ôíöÒå–©Þ disk.nvme0c0n1.in‚ 7.1640625* proc.memory.percent‚17.702078819274902 cpu‚25.674655977145896 disk.rbd1.in‚0 disk.dm-0.out‚ 5744.0625% disk.nvme0c0n1.out‚5106.841796875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2900.30859375 proc.cpu.threads‚17% disk./.usageGB‚1170.5837097167969 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 7.1640625( proc.memory.availableMB‚ 494.58203125) disk./.usagePercent‚76.24710696738622% disk.nvme1c1n1.out‚3288.369140625# disk.nvme1n1.out‚3288.369140625 disk.md0.out‚ 46.359375 disk.dm-0.in‚ 8.62109375# disk.nvme0n1.out‚5106.841796875 disk.rbd0.in‚0$ memory_percent‚96.98131084442139 network.sent‚ 2.510983e+06 network.recv‚434089 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-1.out‚ 2552.37109375™Q—Ùž˜ ‚(:“ ôíöÒ½÷ýÞ gpu.0.gpu‚49.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.793' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.59766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í·E2™ - _wandbruntime‚1552  _runtime‚1552QÚ[`Éš à trainer/global_step‚1649 % _timestamp‚1.7845265886390028e+09 _runtime‚1552.907220148 _step‚132 ' lr-AdamW/pg1‚0.00017699505615446893 ' lr-AdamW/pg2‚0.00017699505615446893²XpmΛ È ' lr-AdamW/pg1‚0.00017699505615446893 ' lr-AdamW/pg2‚0.00017699505615446893 trainer/global_step‚1649 % _timestamp‚1.7845265886390028e+09 _runtime‚1552.907220148 _step‚132„_4š2œ - _wandbruntime‚1552  _runtime‚1552ÀøóèÝ' ×O [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1916132774400 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6839337 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56893289791488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚254374028901376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1915791350272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16316711 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254373670437888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16316711 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252457896126976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250981171456912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1913241409726 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16326854 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252894429905998 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚252894072305364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1916132774400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2883177 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 $ _timestamp‚1.784526588640305e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252457879087616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13353585 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6839057 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1916132774400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2883177 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 $ _timestamp‚1.784526588640305e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252457879087616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13353585 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6839057 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254373670437888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272…ûHåÂ2Ÿ - _wandbruntime‚1553  _runtime‚1553Wdž5  ãM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252606170629120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 % _timestamp‚1.7845265892724044e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13361378 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚254523055038976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1916901449216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16336524 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56927706397184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252606153589760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚254523413509632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚:Ïá0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1917242880512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29687745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253042968961834 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1914009236704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6843100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2884911 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3958189 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3958479 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29697904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252606170629120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1553.540834001 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚254523413509632 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13361380 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56927584620032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251128619163744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1916901449216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16326367 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251128602124384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13361378 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253042611361088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252606153589760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1928517260800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚134 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1928414095872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6843393 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ˆrâî&¡ èM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252606170629120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 % _timestamp‚1.7845265892724044e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13361378 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚254523055038976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1916901449216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16336524 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56927706397184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252606153589760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚254523413509632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1917242880512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29687745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253042968961834 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1914009236704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6843100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2884911 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3958189 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3958479 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29697904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252606170629120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1553.540834001 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚254523413509632 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13361380 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56927584620032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251128619163744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1916901449216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16326367 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251128602124384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13361378 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253042611361088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252606153589760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1928517260800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚134 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1928414095872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6843393 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0†¿ÞÎ2¢ - _wandbruntime‚1553  _runtime‚1553•‚5@£ ÿ 0loss/FRt1/assignment_loss‚1.0153934178528963 /loss/FRt2/assignment_loss‚2.541339895483845 0loss/FBt2/assignment_loss‚0.6128800747580064 /loss/FRt1/detection_loss‚1.3630838835508459 /loss/FRt2/detection_loss‚0.5722955901457418 1loss/SRqqt2/detection_loss‚1.1276366953043395 .loss/FBt1/detection_loss‚1.184610765755526 % _timestamp‚1.7845265892736704e+09 1loss/FBt1/assignment_loss‚0.06404201855234344 /loss/FBt2/detection_loss‚1.2317636641471281 epoch‚4 _runtime‚1553.541583492 2loss/SRqqt1/assignment_loss‚0.6579359504685272 2loss/SRqqt2/assignment_loss‚0.7829592295627444 1loss/SRqqt1/detection_loss‚0.7175025920500392 &loss/total_loss‚11.871443777631985 trainer/global_step‚1649 _step‚135ƒÒ•” ¤ „ 1loss/FBt1/assignment_loss‚0.06404201855234344 /loss/FBt2/detection_loss‚1.2317636641471281 epoch‚4 _runtime‚1553.541583492 2loss/SRqqt1/assignment_loss‚0.6579359504685272 2loss/SRqqt2/assignment_loss‚0.7829592295627444 1loss/SRqqt1/detection_loss‚0.7175025920500392 &loss/total_loss‚11.871443777631985 trainer/global_step‚1649 _step‚135 0loss/FRt1/assignment_loss‚1.0153934178528963 /loss/FRt2/assignment_loss‚2.541339895483845 0loss/FBt2/assignment_loss‚0.6128800747580064 /loss/FRt1/detection_loss‚1.3630838835508459 /loss/FRt2/detection_loss‚0.5722955901457418 1loss/SRqqt2/detection_loss‚1.1276366953043395 .loss/FBt1/detection_loss‚1.184610765755526 % _timestamp‚1.7845265892736704e+09‡xBﶉ¥ ‚(:þ ƒîöÒã·ôÝ disk.dm-0.out‚ 5752.328125& disk.nvme0c0n1.out‚5115.0791015625 disk.nvme0n1.in‚ 7.1640625$ disk.nvme0n1.out‚5115.0791015625$ memory_percent‚96.98305130004883 network.sent‚ 2.515876e+06" disk.dm-1.out‚2552.56396484375 disk.nvme0c0n1.in‚ 7.1640625# proc.memory.rssMB‚ 2900.30859375 cpu‚25.67253231541301' disk.nvme1c1n1.out‚3289.22119140625 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.md0.out‚46.65625 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.702078819274902 proc.cpu.threads‚17 network.recv‚437937) disk./.usagePercent‚76.23164716284819$ disk./.usageGB‚1170.346363067627 disk.rbd1.in‚0 disk.md0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 494.296875 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1n1.out‚3289.22119140625 disk.dm-0.in‚ 8.62109375Û+rœ¦ ‚(:‘ ƒîöÒƒªèÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚256.4' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.46666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ët­qž§ ‚(:“ ’îöÒÛøÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚285.584' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.19466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ⲻ–¨ ‚(:‹ ’îöÒàÆêÞ disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚3291.72119140625 disk.rbd1.out‚0$ memory_percent‚96.98150157928467 network.sent‚ 2.580007e+06) disk./.usagePercent‚76.23164865369226 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 disk.dm-0.in‚ 8.62109375 disk.nvme0n1.in‚ 7.1640625( proc.memory.availableMB‚ 494.55078125# proc.memory.rssMB‚ 2900.30859375% disk./.usageGB‚1170.3463859558105% disk.nvme1n1.out‚3291.72119140625 disk.md0.out‚ 47.0078125" disk.dm-1.out‚2552.65771484375 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 network.recv‚441039 disk.dm-1.in‚0' disk.nvme0c0n1.out‚5126.18408203125* proc.memory.percent‚17.702078819274902 cpu‚25.670403500102815 proc.cpu.threads‚17" disk.dm-0.out‚5765.08154296875 disk.nvme0c0n1.in‚ 7.1640625% disk.nvme0n1.out‚5126.18408203125·Ò7© ‚(:„ ¡îöÒݬ²Þ disk.nvme0n1.in‚ 7.1640625$ disk.nvme0n1.out‚5127.3662109375 disk.rbd0.in‚0' disk.nvme1c1n1.out‚3292.56201171875 disk.md0.out‚ 47.609375% disk./.usageGB‚1170.3451957702637" disk.dm-0.out‚5765.69091796875 disk.nvme0c0n1.in‚ 7.1640625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125* proc.memory.percent‚17.702078819274902 network.recv‚444264 disk.dm-0.in‚ 8.62109375 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚3292.56201171875( proc.memory.availableMB‚ 494.55078125# proc.memory.rssMB‚ 2900.30859375( disk./.usagePercent‚76.2315711298006& disk.nvme0c0n1.out‚5127.3662109375 disk.md0.in‚0$ memory_percent‚96.98150157928467 cpu‚25.668360065397636 proc.cpu.threads‚17 network.sent‚ 2.584135e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 2552.80078125¥ì%U‘ª ‚(:† ¡îöÒ²Á÷Þ gpu.0.gpu‚5.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚275.724' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ñ™22« - _wandbruntime‚1597  _runtime‚1597›´ÑFɬ à ' lr-AdamW/pg1‚0.00017684978034944012 ' lr-AdamW/pg2‚0.00017684978034944012 trainer/global_step‚1699 % _timestamp‚1.7845266336611483e+09 _runtime‚1597.929321854 _step‚136if7Sέ È ' lr-AdamW/pg1‚0.00017684978034944012 ' lr-AdamW/pg2‚0.00017684978034944012 trainer/global_step‚1699 % _timestamp‚1.7845266336611483e+09 _runtime‚1597.929321854 _step‚136ˆ¯h2í2® - _wandbruntime‚1597  _runtime‚1597Ç ›ß'¯ ÙO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚1597.930441013 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30544598 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845266336622996e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30554743 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261842964646912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258352721869872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260321088892372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1971360452096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13743237 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259871621234176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261842964646912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259871604194816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58608132994048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16811506 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13743237 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1971360452096 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58608261069824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1983129235968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260321446493006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232Öhµ&° ÞO M@DeviceStatsMonitor.on_train_¼Põ¾'batch_start/allocation.all.allocated‚30554743 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261842964646912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258352721869872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260321088892372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1971360452096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13743237 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259871621234176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261842964646912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259871604194816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58608132994048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16811506 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13743237 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1971360452096 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58608261069824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1983129235968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260321446493006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚1597.930441013 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30544598 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845266336622996e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488‰‡u "2± - _wandbruntime‚1598  _runtime‚1598¥q’÷é&² ãM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260019878696960 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7044769 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7044475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16821176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16811019 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258500135497984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13751030 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261992707718656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2971614 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30572206 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2971611 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13751028 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1969833011498 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30562047 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1972811982336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚1598.561733175 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260019878696960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4072864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1984246641152 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56658202640384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260019895736320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258500152537344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚138 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1972811982336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1972470551040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261992349248000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260019895736320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13751030 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16811019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58642556640768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845266342934272e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261992707718656­þî&³ èM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30562047 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58642430669312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1984354000384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16821176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16811019 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258500135497984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13751030 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261992707718656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2971614 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30572206 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2971611 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13751028 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1969833011498 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30562047 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1972811982336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚1598.561733175 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260019878696960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4072864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1984246641152 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56658202640384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260019895736320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258500152537344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚138 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1972811982336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1972470551040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261992349248000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260019895736320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13751030 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16811019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58642556640768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845266342934272e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261992707718656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260019878696960 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7044769 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7044475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269312258855936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267285346341376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2027270978048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31418900 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267285346341376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267285329302016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17296158 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60323116499456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3056580 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267748105479380 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267285329302016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17286015 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17286015 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4185271 _runtime‚1642.767422082 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14132887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4185562 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2039080625152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60322996812288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265724255243472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265724238204112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269312258855936 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚1749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269312617319424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2026929553920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534®Gž5ä' ÞO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31418900 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267285346341376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267285329302016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17296158 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60323116499456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3056580 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267748105479380 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267285329302016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17286015 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17286015 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4185271 _runtime‚1642.767422082 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14132887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4185562 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2039080625152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60322996812288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265724255243472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265724238204112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269312258855936 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚1749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269312617319424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2026929553920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267748463080014 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2023867275268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7241848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14132885 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14132887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2024207836542 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269312617319424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14132885 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7242142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58284017262080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2026929553920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845266784992685e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17296158 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31429045 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2038979550208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269312258855936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267285346341376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2027270978048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344æ•Ý2à - _wandbruntime‚1643  _runtime‚1643 sݸâ!Ä ãM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269462001927680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2028039652864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267896644535104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269461643457024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58317216144384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2025316224906 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17295671 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845266791300006e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14140678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2040201005056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265871668871584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2024975663520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4187497 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265871685910944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60357289080832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14140678 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17305828 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4187792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7245808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58317197532160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2028381084160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267433603804160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2028039652864å¶ú@î&Å èM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14140678 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17305828 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4187792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7245808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58317197532160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2028381084160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267433603804160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2028039652864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269462001927680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2028039652864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267896644535104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269461643457024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58317216144384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2025316224906 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17295671 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845266791300006e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14140678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2040201005056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265871668871584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2024975663520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4187497 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265871685910944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60357289080832Žz vÆ2Æ - _wandbruntime‚1643  _runtime‚1643m£íÇ € 1loss/FBt1/assignment_loss‚0.06436564904324472 0loss/FBt2/assignment_loss‚0.4207951467342622 &loss/total_loss‚13.171105618885893 epoch‚4 0loss/FRt2/assignment_loss‚2.7469538119250694 2loss/SRqqt2/assignment_loss‚0.8274534568996527 /loss/FRt2/detection_loss‚0.6094983889609554 1loss/SRqqt2/detection_loss‚1.2562927663915666 trainer/global_step‚1749 .loss/FRt1/detection_loss‚1.364164684396837 /loss/FBt1/detection_loss‚2.2026357079507415 % _timestamp‚1.7845266791311202e+09 _step‚143 1loss/SRqqt1/detection_loss‚0.6752984302704766 /loss/FBt2/detection_loss‚1.0484151441384764 _runtime‚1643.399162875 0loss/FRt1/assignment_loss‚1.2381976769307432 2loss/SRqqt1/assignment_loss‚0.7170347552438666À èñ È … 0loss/FRt1/assignment_loss‚1.2381976769307432 2loss/SRqqt1/assignment_loss‚0.7170347552438666 1loss/FBt1/assignment_loss‚0.06436564904324472 0loss/FBt2/assignment_loss‚0.4207951467342622 &loss/total_loss‚13.171105618885893 epoch‚4 0loss/FRt2/assignment_loss‚2.7469538119250694 2loss/SRqqt2/assignment_loss‚0.8274534568996527 /loss/FRt2/detection_loss‚0.6094983889609554 1loss/SRqqt2/detection_loss‚1.2562927663915666 trainer/global_step‚1749 .loss/FRt1/detection_loss‚1.364164684396837 /loss/FBt1/detection_loss‚2.2026357079507415 % _timestamp‚1.7845266791311202e+09 _step‚143 1loss/SRqqt1/detection_loss‚0.6752984302704766 /loss/FBt2/detection_loss‚1.0484151441384764 _runtime‚1643.399162875bÀ´,ŽÉ ‚(:ƒ ÝîöÒíÛÂÞ proc.cpu.threads‚17 network.recv‚458338( disk./.usagePercent‚76.2470982707958& disk.nvme0c0n1.out‚5348.4189453125 disk.nvme0n1.in‚ 7.1640625 disk.nvme1n1.in‚ 1.45703125 cpu‚25.667244836190886 network.sent‚ 2.67245e+06$ disk.nvme0n1.out‚5348.4189453125 disk.rbd0.in‚0& disk.nvme1c1n1.out‚3324.0693359375 disk.md0.out‚ 49.015625 disk.dm-0.in‚ 8.62109375$ memory_percent‚96.98865413665771" disk.dm-1.out‚2553.35302734375 disk.nvme0c0n1.in‚ 7.1640625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0( proc.memory.availableMB‚ 493.37890625) proc.memory.percent‚17.70789623260498% disk./.usageGB‚1170.5835762023926 disk.dm-1.in‚0 disk.rbd1.out‚0$ disk.nvme1n1.out‚3324.0693359375 disk.md0.in‚0" disk.dm-0.out‚6014.64794921875# proc.memory.rssMB‚ 2901.26171875·]ÃcŸÊ ‚(:” ÝîöÒ·ÊüÞ gpu.0.gpu‚35.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.031' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.010333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ÝRW…Ë ‚(:ú ìîöÒб›Þ disk.dm-0.in‚ 8.62109375 disk.nvme0n1.in‚ 7.1640625' disk.nvme1c1n1.out‚3324.60009765625 disk.rbd1.in‚0% disk.nvme1n1.out‚3324.60009765625$ proc.memory.availableMB‚494.3125 cpu‚25.670520639545195 proc.cpu.threads‚17 network.sent‚ 2.724113e+06 network.recv‚462713) disk./.usagePercent‚76.24710721586024! disk.dm-0.out‚6017.7958984375" disk.dm-1.out‚2553.42724609375 disk.nvme0c0n1.in‚ 7.1640625# proc.memory.rssMB‚ 2900.30859375* proc.memory.percent‚17.702078819274902% disk./.usageGB‚1170.5837135314941 disk.md0.out‚49.375 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.out‚ 5351.8984375! disk.nvme0n1.out‚ 5351.8984375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0$ memory_percent‚96.98295593261719d¤ø+Ì ‚(:’ ìîöÒ¾¯ÛÞ gpu.0.gpu‚32.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.615' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.20500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ ‚Í ‚(:÷ ûîöÒÃΓÞ disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.62109375! disk.dm-0.out‚6018.3193359375 disk.dm-1.in‚0& proc.memory.availableMB‚ 493.328125) proc.memory.percent‚17.70789623260498 cpu‚25.67387812907766$ disk./.usageGB‚1170.583236694336 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.out‚ 3325.046875 disk.rbd0.out‚0) disk./.usagePercent‚76.24707615660876" disk.nvme1c1n1.out‚ 3325.046875 disk.md0.in‚0 disk.nvme0c0n1.in‚ 7.1640625# proc.memory.rssMB‚ 2901.26171875 proc.cpu.threads‚17 network.recv‚466203 disk.md0.out‚ 49.63671875 disk.dm-1.out‚ 2553.49609375& disk.nvme0c0n1.out‚5352.6044921875 disk.nvme0n1.in‚ 7.1640625$ disk.nvme0n1.out‚5352.6044921875 disk.rbd0.in‚0$ memory_percent‚96.98896408081055 network.sent‚ 2.727035e+06@*”žÎ ‚(:“ ûîöÒ›ÏîÞ gpu.0.gpu‚76.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.064' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.02133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FN×Ö2Ï - _wandbruntime‚1688  _runtime‚1688>,Ê»ÈÐ Â ' lr-AdamW/pg1‚0.00017654238633454375 ' lr-AdamW/pg2‚0.00017654238633454375 trainer/global_step‚1799 % _timestamp‚1.7845267242641332e+09 _runtime‚ 1688.53227329 _step‚144•¢ª—ÍÑ Ç ' lr-AdamW/pg1‚0.00017654238633454375 ' lr-AdamW/pg2‚0.00017654238633454375 trainer/global_step‚1799 % _timestamp‚1.7845267242641332e+09 _runtime‚ 1688.53227329 _step‚144¹f$02Ò - _wandbruntime‚1688  _runtime‚16882Êîêß'Ó ÙO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274699071448576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274699054409216 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62037866078720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273095771577712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275175479667022 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59943030766080 _runtime‚1688.533064301 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276781553064960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276781553064960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2082498655744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32303347 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2094835312640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2079350488676 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17780810 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7443065 % _timestamp‚1.7845267242650177e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276781911528448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4299785 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2094938484736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7442768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17770667 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2082840079872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273095788617072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274699054409216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3143277 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2082840079872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275175122066388 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14522537 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2079691049950 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274699071448576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 trainer/global_step‚1799 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32293202 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14522535 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3143280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14522535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14522537 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4299491 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59943049378304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17770667 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚276781911528448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320rÙ«J Ô ÞO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17780810 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2082498655744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276781553064960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2082498655744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32303347 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2094835312640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2079350488676 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17780810 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7443065 % _timestamp‚1.7845267242650177e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276781911528448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4299785 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2094938484736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7442768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17770667 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2082840079872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273095788617072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274699054409216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3143277 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2082840079872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275175122066388 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_ν±¯Úpool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14522537 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2079691049950 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274699071448576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 trainer/global_step‚1799 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32293202 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14522535 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3143280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14522535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14522537 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4299491 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59943049378304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17770667 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚276781911528448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274699071448576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274699054409216 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62037866078720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273095771577712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275175479667022 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59943030766080 _runtime‚1688.533064301 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276781553064960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274847328911360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2080458876928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4301732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276931296136704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2083608754688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273243202245184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2080799438314 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275323661122112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273243219284544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59976229648384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14530328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7446743 trainer/global_step‚1799 _runtime‚1689.166929839 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276931296136704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17790480 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275324018722858 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3145011 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2083950185984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845267248985596e+09 _step‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2083608754688 f—˜î&× èM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14530328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7446743 trainer/global_step‚1799 _runtime‚1689.166929839 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276931296136704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17790480 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275324018722858 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3145011 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2083950185984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845267248985596e+09 _step‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2083608754688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17780323 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274847328911360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4302014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7447028 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3145014 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276930937666048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274847328911360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2080458876928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4301732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276931296136704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2083608754688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273243202245184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2080799438314 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275323661122112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273243219284544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59976229648384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0’XbF2Ø - _wandbruntime‚1689  _runtime‚1689p¹^0Ù ÿ 0loss/FRt2/assignment_loss‚2.6741865026042064 2loss/SRqqt2/assignment_loss‚0.7158001422151872 /loss/FRt1/detection_loss‚1.3829678567410735 % _timestamp‚1.7845267249001591e+09 1loss/FBt2/assignment_loss‚0.28963828470472935 1loss/SRqqt1/detection_loss‚0.6903428778266727 .loss/FBt2/detection_loss‚1.017945162117358 0loss/FRt1/assignment_loss‚1.0919397452236277 1loss/SRqqt2/detection_loss‚1.1002152260326588 .loss/FBt1/detection_loss‚0.769150905461787 &loss/total_loss‚11.076364767447735 epoch‚4 2loss/SRqqt1/assignment_loss‚0.6048865914783214 0loss/FBt1/assignment_loss‚0.1266897846922758 /loss/FRt2/detection_loss‚0.6126016883498382 trainer/global_step‚1799 _runtime‚1689.168150161 _step‚147¬Ò` Ú „ % _timestamp‚1.7845267249001591e+09 1loss/FBt2/assignment_loss‚0.28963828470472935 1loss/SRqqt1/detection_loss‚0.6903428778266727 .loss/FBt2/detection_loss‚1.017945162117358 0loss/FRt1/assignment_loss‚1.0919397452236277 1loss/SRqqt2/detection_loss‚1.1002152260326588 .loss/FBt1/detection_loss‚0.769150905461787 &loss/total_loss‚11.076364767447735 epoch‚4 2loss/SRqqt1/assignment_loss‚0.6048865914783214 0loss/FBt1/assignment_loss‚0.1266897846922758 /loss/FRt2/detection_loss‚0.6126016883498382 trainer/global_step‚1799 _runtime‚1689.168150161 _step‚147 0loss/FRt2/assignment_loss‚2.6741865026042064 2loss/SRqqt2/assignment_loss‚0.7158001422151872 /loss/FRt1/detection_loss‚1.3829678567410735“î:%y‡Û ‚(:ü ŠïöÒ˜¼áÝ disk.dm-1.out‚ 2553.5703125 disk.nvme0c0n1.in‚ 7.1640625 disk.rbd1.out‚0 network.sent‚ 2.733018e+06) disk./.usagePercent‚76.24707814440085 disk.md0.in‚0! disk.dm-0.out‚6028.3935546875 disk.nvme0n1.in‚ 7.1640625 cpu‚25.677479302878474# disk.nvme0c0n1.out‚ 5361.1640625! disk.nvme0n1.out‚ 5361.1640625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125& disk.nvme1c1n1.out‚3327.1064453125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.98355197906494 network.recv‚469692 disk.dm-0.in‚ 8.62109375 disk.rbd0.out‚0$ disk.nvme1n1.out‚3327.1064453125( proc.memory.availableMB‚ 494.21484375# proc.memory.rssMB‚ 2900.30859375* proc.memory.percent‚17.702078819274902 proc.cpu.threads‚17$ disk./.usageGB‚1170.583267211914 disk.md0.out‚ 49.84765625 disk.dm-1.in‚0ÿŸMËÜ ‚(:’ ŠïöÒ¤ó×Þ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.629' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.876333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t~ð‰Ý ‚(:þ ™ïöÒ…ôÝ$ memory_percent‚96.98405265808105# proc.memory.rssMB‚ 2900.30859375 proc.cpu.threads‚17 network.recv‚474265) disk./.usagePercent‚76.24707342339462 disk.nvme0c0n1.in‚ 7.1640625 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125 cpu‚25.680421850591017$ disk./.usageGB‚1170.583194732666 disk.dm-0.in‚ 8.62109375! disk.dm-0.out‚6029.0888671875 disk.dm-1.out‚2553.642578125 disk.nvme0n1.in‚ 7.1640625$ disk.nvme1c1n1.out‚ 3327.68359375 disk.rbd1.out‚0* proc.memory.percent‚17.702078819274902 disk.md0.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚5362.0830078125$ disk.nvme0n1.out‚5362.0830078125 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.nvme1n1.out‚ 3327.68359375' proc.memory.availableMB‚ 494.1328125 network.sent‚ 2.794365e+06 disk.md0.out‚ 50.1796875 disk.rbd0.out‚0S>üoÞ ‚(:’ ™ïöÒ®ùÏÞ gpu.0.gpu‚58.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚168.9' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚56.300000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02·¾ü‘ß ‚(:† ¨ïöÒÉ…²Þ cpu‚25.683590370308224 proc.cpu.threads‚17 disk.dm-0.in‚ 8.62109375 disk.dm-1.in‚0 disk.dm-1.out‚2553.681640625) disk./.usagePercent‚76.24712386361901$ disk./.usageGB‚1170.583969116211 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.in‚0 network.sent‚ 2.801261e+06 disk.nvme1n1.in‚ 1.45703125" disk.dm-0.out‚6030.99169921875 disk.md0.out‚ 50.36328125 disk.nvme0c0n1.in‚ 7.1640625' disk.nvme0c0n1.out‚5363.94775390625 disk.nvme0n1.in‚ 7.1640625% disk.nvme0n1.out‚5363.94775390625 network.recv‚477613$ disk.nvme1n1.out‚3328.1689453125 disk.rbd0.out‚0$ memory_percent‚96.98290824890137# proc.memory.rssMB‚ 2900.11328125* proc.memory.percent‚17.700886726379395 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125& disk.nvme1c1n1.out‚3328.1689453125' proc.memory.availableMB‚ 494.3203125æÞfÝžà ‚(:“ ¨ïöÒ³×ðÞ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚264.684' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.22800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rm^2á - _wandbruntime‚1733  _runtime‚1733hÈŸÜÉâ à % _timestamp‚1.7845267696879938e+09 _runtime‚1733.956138605 _step‚148 ' lr-AdamW/pg1‚0.00017638028786801937 ' lr-AdamW/pg2‚0.00017638028786801937 trainer/global_step‚1849½žÎã È ' lr-AdamW/pg2‚0.00017638028786801937 trainer/global_step‚1849 % _timestamp‚1.7845267696879938e+09 _runtime‚1733.956138605 _step‚148 ' lr-AdamW/pg1‚0.00017638028786801937”ÎôõÙ2ä - _wandbruntime‚1733  _runtime‚1733EÆ­Uêå ÙO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284251205737472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2134833702084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282112779516416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2138409181696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1733.957241487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14912187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2138067757568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/requested_byteÃ^íõ!s.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14912187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414853 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63752882557440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282602496254030 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18255319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280467321990672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280467304951312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282112796555776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3229980 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3229977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7644833 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282602138653396 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18265462 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33167504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63752760773120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61602044270080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282112796555776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14912185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7644534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2150819675136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18255319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚149 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33177649 % _timestamp‚1.7845267696891162e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18265462 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2138067757568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2150716503040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2135174263358 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284250847273984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282112779516416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284250847273984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4414557 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634'?ä'æ ÞO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33167504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63752760773120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61602044270080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282112796555776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14912185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7644534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2150819675136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18255319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚149 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33177649 % _timestamp‚1.7845267696891162e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18265462 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2138067757568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2150716503040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2135174263358 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284250847273984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282112779516416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284250847273984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4414557 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284251205737472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2134833702084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282112779516416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2138409181696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1733.957241487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14912187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2138067757568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14912187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414853 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63752882557440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282602496254030 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18255319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280467321990672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280467304951312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282112796555776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3229980 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3229977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7644833 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282602138653396 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18265462 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0•Š°Lì2ç - _wandbruntime‚1734  _runtime‚1734ý¬ 8é&è ãM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14919980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3231714 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3231711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚1849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2139519287808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18275132 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7648595 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280614752658144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2139177856512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282751035309866 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2135942090336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18275132 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280614735618784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2136282651722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61635224540160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18264975 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282261071057920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14919980 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63787181429760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18264975 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33184953 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2139519287808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2151938277376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14919978 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33184953 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282750677709120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63787057555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4417153 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2151833015296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282261054018560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284400590345728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282261054018560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61635243152384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1734.589625284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2139177856512÷D^éé èM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284400231875072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282261071057920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4416884 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282261071057920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14919980 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63787181429760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18264975 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33184953 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2139519287808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2151938277376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14919978 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33184953 ;4DeviceStatsMonitor.on_train_batch_end/max_split_siûÆÇaze‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282750677709120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63787057555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4417153 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2151833015296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282261054018560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284400590345728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282261054018560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61635243152384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1734.589625284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2139177856512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14919980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3231714 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3231711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚1849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2139519287808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18275132 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7648595 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280614752658144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2139177856512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282751035309866 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2135942090336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18275132 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280614735618784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2136282651722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61635224540160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18264975 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18750114 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15301835 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18739971 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65467739890176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2193636859392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15301837 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18750114 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65467616008704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290029512841038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34051951 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7846017 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2193978283520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4529340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7846306 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291720141483008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289526504623616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚291720499946496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63261057774080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287838838324912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15301835 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845268153417292e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15301837 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3316680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34041806 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4529626 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _step‚153 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63261076386304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2206558234624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2193978283520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289526521662976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291720499946496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287838855364272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2190316915492 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2206663503872`.\)ä'ø ÞO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18739971 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2193636859392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15301837 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18750114 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65467616008704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290029512841038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34051951 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7846017 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2193978283520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4529340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7846306 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291720141483008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289526504623616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚291720499946496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63261057774080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287838838324912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15301835 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845268153417292e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15301837 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3316680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34041806 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4529626 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _step‚153 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63261076386304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2206558234624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2193978283520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289526521662976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291720499946496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287838855364272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2190316915492 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2206663503872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289526521662976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290029155240404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18750114 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15301835 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18739971 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65467739890176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291869884554752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3318414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65502034347008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2191765865130 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚291869526084096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7850051 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2191425303744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18749627 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚291869884554752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18759784 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7850341 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3318411 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4531640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290177694296128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289674779125760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34059255 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2194746958336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63294256656384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63294238044160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2195088389632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18749627 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290178051896874 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289674779125760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15309630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65501912569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2207674525696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2195088389632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287986286031744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2207777690624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232¯É“$î&û èM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289674779125760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15309630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65501912569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2207674525696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2195088389632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287986286031744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2207777690624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 trainer/global_step‚1899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291869884554752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3318414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65502034347008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2191765865130 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚291869526084096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7850051 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2191425303744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18749627 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚291869884554752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18759784 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7850341 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3318411 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4531640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290177694296128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289674779125760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34059255 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2194746958336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63294256656384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63294238044160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2195088389632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18749627 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290178051896874 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0šƒ k2ü - _wandbruntime‚1780  _runtime‚1780iõì¹ý ÿ 0loss/FBt2/assignment_loss‚0.2321373486031137 /loss/FRt2/detection_loss‚0.5914162495885085 0loss/SRqqt1/detection_loss‚0.690281281192971 /loss/FBt1/detection_loss‚0.7095406228448371 trainer/global_step‚1899 _step‚155 2loss/SRqqt1/assignment_loss‚0.6405198552040064 epoch‚4 0loss/FRt1/assignment_loss‚1.1086974755311623 0loss/FRt2/assignment_loss‚2.5942311764956845 1loss/FBt1/assignment_loss‚0.10762196761419573 % _timestamp‚1.7845268159727373e+09 _runtime‚1780.240539878 2loss/SRqqt2/assignment_loss‚0.7466732013290143 .loss/FRt1/detection_loss‚1.351732849592476 1loss/SRqqt2/detection_loss‚1.1172986951982458 /loss/FBt2/detection_loss‚1.5049063270645482 &loss/total_loss‚11.395057050258764íMX9 þ „ /loss/FBt1/detection_loss‚0.7095406228448371 trainer/global_step‚1899 _step‚155 2loss/SRqqt1/assignment_loss‚0.6405198552040064 epoch‚4 0loss/FRt1/assignment_loss‚1.1086974755311623 0loss/FRt2/assignment_loss‚2.5942311764956845 1loss/FBt1/assignment_loss‚0.10762196761419573 % _timestamp‚1.7845268159727373e+09 _runtime‚1780.240539878 2loss/SRqqt2/assignment_loss‚0.7466732013290143 .loss/FRt1/detection_loss‚1.351732849592476 1loss/SRqqt2/detection_loss‚1.1172986951982458 /loss/FBt2/detection_loss‚1.5049063270645482 &loss/total_loss‚11.395057050258764 0loss/FBt2/assignment_loss‚0.2321373486031137 /loss/FRt2/detection_loss‚0.5914162495885085 0loss/SRqqt1/detection_loss‚0.690281281192971›øí ãˆÿ ‚(:ý äïöÒ‘˜¦Þ% disk.nvme1c1n1.out‚3333.095703125 disk.md0.out‚ 51.75390625 disk.dm-1.in‚0 disk.dm-1.out‚2554.013671875 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.889275e+06% disk./.usageGB‚1164.9111938476562 disk.md0.in‚0 disk.dm-0.out‚6055.462890625 disk.nvme0c0n1.in‚ 7.1640625 disk.nvme0n1.in‚ 7.1640625( proc.memory.availableMB‚ 493.19140625) proc.memory.percent‚17.70789623260498( disk./.usagePercent‚75.8776221363072 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125# disk.nvme1n1.out‚3333.095703125 disk.dm-0.in‚ 8.62109375% disk.nvme0c0n1.out‚5386.865234375# proc.memory.rssMB‚ 2901.26171875 network.recv‚491632 disk.rbd1.out‚0# disk.nvme0n1.out‚5386.865234375 disk.rbd0.in‚0# memory_percent‚96.9897985458374 cpu‚25.695381221755095í“Tl“€ ‚(:ˆ äïöÒņèÞ gpu.0.gpu‚72.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.051' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.017 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š¹/’ ‚(:‡ óïöÒùÅéÝ disk.rbd1.out‚0% disk.nvme1n1.out‚3333.59033203125 disk.nvme0c0n1.in‚ 7.1640625% disk.nvme0n1.out‚5388.35498046875# proc.memory.rssMB‚ 2900.30859375 proc.cpu.threads‚17 network.recv‚496253) disk./.usagePercent‚75.87764201422814 disk.dm-0.in‚ 8.62109375" disk.dm-0.out‚6056.71337890625" disk.dm-1.out‚2554.09716796875' disk.nvme0c0n1.out‚5388.35498046875 disk.nvme0n1.in‚ 7.1640625 disk.rbd0.in‚0 network.sent‚ 2.938916e+06% disk./.usageGB‚1164.9114990234375 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 disk.rbd0.out‚0& proc.memory.availableMB‚ 494.390625* proc.memory.percent‚17.702078819274902 cpu‚25.69818614935788 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚3333.59033203125 disk.md0.in‚0 disk.md0.out‚ 52.046875 disk.dm-1.in‚0$ memory_percent‚96.98247909545898¡µxèž‚ ‚(:“ óïöÒòºÀÞ gpu.0.gpu‚87.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.136' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.37866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âr<àƒ ‚(:‚ ‚ðöÒóŠœÞ disk.dm-0.in‚ 8.62109375' disk.nvme0c0n1.out‚5551.71630859375% disk.nvme0n1.out‚5551.71630859375% disk.nvme1c1n1.out‚5278.892578125 disk.rbd1.in‚0 cpu‚25.701056112418424 proc.cpu.threads‚17) disk./.usagePercent‚76.07300048192032% disk./.usageGB‚1167.9107398986816! disk.dm-0.out‚8163.8896484375 disk.nvme0c0n1.in‚ 7.1640625 disk.rbd0.in‚0$ memory_percent‚96.99037075042725 network.recv‚499616 disk.md0.out‚52.6875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 7.1640625 disk.rbd1.out‚0# proc.memory.rssMB‚ 2901.26171875 disk.nvme1n1.in‚ 1.45703125# disk.nvme1n1.out‚5278.892578125 disk.md0.in‚0" disk.dm-1.out‚2554.22998046875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125( proc.memory.availableMB‚ 493.09765625) proc.memory.percent‚17.70789623260498 network.sent‚ 2.943761e+06‰…ïoœ„ ‚(:‘ ‚ðöÒÒƒêÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚262.565' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.52166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§õ:Û2… - _wandbruntime‚1824  _runtime‚1824F”Y0Ȇ  ' lr-AdamW/pg1‚0.00017603934007255753 ' lr-AdamW/pg2‚0.00017603934007255753 trainer/global_step‚1949 $ _timestamp‚1.784526860321067e+09 _runtime‚1824.589195113 _step‚156*¶½Í‡ Ç trainer/global_step‚1949 $ _timestamp‚1.784526860321067e+09 _runtime‚1824.589195113 _step‚156 ' lr-AdamW/pg1‚0.00017603934007255753 ' lr-AdamW/pg2‚0.00017603934007255753œ%v¥2ˆ - _wandbruntime‚1824  _runtime‚1824¨ó7êÚ‰ ÙO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19224623 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64920089890304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64920071278080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8047203 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262546673664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262441404416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295210388737872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299189794155520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3403377 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2246140690174 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2249205961216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚1824.591504827 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15691487 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4643826 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2249205961216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297456529428046 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19234766 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299189435692032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15691487 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296940246770176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15691485 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1949 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299189435692032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296940229730816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34926253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _step‚157 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296940229730816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8047490 =6DeviceStatsMonitor.on_train_babD°}tch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15691485 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296940246770176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2249547385344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚297456171827412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2245800128900 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19224623 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299189794155520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4644110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295210371698512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67182512682496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272*3¬Öä'Š ÞO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19224623 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64920089890304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64920071278080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8047203 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262546673664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262441404416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295210388737872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299189794155520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3403377 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2246140690174 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2249205961216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚1824.591504827 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15691487 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4643826 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2249205961216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297456529428046 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19234766 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299189435692032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15691487 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296940246770176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15691485 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1949 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299189435692032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296940229730816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34926253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _step‚157 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296940229730816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8047490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15691485 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296940246770176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2249547385344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚297456171827412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2245800128900 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19224623 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299189794155520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4644110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295210371698512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67182512682496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272}ý2‹ - _wandbruntime‚1825  _runtime‚1825}eðÅé&Œ ãM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295357819405344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297088504232960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34933557 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15699278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297605068483882 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297088504232960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2263663801856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2250316060160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3405114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15699280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3405111 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299339178763776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67216933962240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19244436 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚1949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚297088521272320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚1825.546898693 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8051491 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8051204 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚297604710883136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚158 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚299338820293120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2247249078538 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64953251548160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34943716 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19244436 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚297088521272320/6]Ù" èM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚1949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚297088521272320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚1825.546898693 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8051491 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8051204 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚297604710883136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚158 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚299338820293120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2247249078538 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64953251548160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34943716 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19244436 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚297088521272320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295357819405344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297088504232960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34933557 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15699278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297605068483882 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297088504232960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2263663801856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2250316060160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3405114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15699280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3405111 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299339178763776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67216933962240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19244436 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_èǘ{train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528žä$"2Ž - _wandbruntime‚1825  _runtime‚1825H'à € /loss/FRt2/assignment_loss‚2.699175398891264 1loss/FBt1/assignment_loss‚0.10710704052963442 1loss/SRqqt1/detection_loss‚0.7178556439662762 epoch‚4 2loss/SRqqt1/assignment_loss‚0.6228494329328045 1loss/FBt2/assignment_loss‚0.10904764979391192 /loss/FRt2/detection_loss‚0.6062308155324171 /loss/FBt1/detection_loss‚0.8445018488416278 &loss/total_loss‚11.452739778485235 % _timestamp‚1.7845268612796798e+09 0loss/FRt1/assignment_loss‚1.0266140360698521 1loss/SRqqt2/detection_loss‚1.2452468239623358 _runtime‚1825.547495414 _step‚159 2loss/SRqqt2/assignment_loss‚0.7373428327660866 .loss/FRt1/detection_loss‚1.375591084693132 /loss/FBt2/detection_loss‚1.3611771705058904 trainer/global_step‚1949ݾ Ô  … .loss/FRt1/detection_loss‚1.375591084693132 /loss/FBt2/detection_loss‚1.3611771705058904 trainer/global_step‚1949 /loss/FRt2/assignment_loss‚2.699175398891264 1loss/FBt1/assignment_loss‚0.10710704052963442 1loss/SRqqt1/detection_loss‚0.7178556439662762 epoch‚4 2loss/SRqqt1/assignment_loss‚0.6228494329328045 1loss/FBt2/assignment_loss‚0.10904764979391192 /loss/FRt2/detection_loss‚0.6062308155324171 /loss/FBt1/detection_loss‚0.8445018488416278 &loss/total_loss‚11.452739778485235 % _timestamp‚1.7845268612796798e+09 0loss/FRt1/assignment_loss‚1.0266140360698521 1loss/SRqqt2/detection_loss‚1.2452468239623358 _runtime‚1825.547495414 _step‚159 2loss/SRqqt2/assignment_loss‚0.7373428327660866Ÿ¦!‹¿‘ ‚(:‚ ‘ðöÒÅò¾Þ$ memory_percent‚96.98894023895264 network.recv‚503285% disk.nvme1n1.out‚5281.43994140625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 7.1640625 disk.nvme0n1.in‚ 7.1640625# disk.nvme0n1.out‚5607.244140625 disk.nvme1c1n1.in‚ 1.45703125( proc.memory.availableMB‚ 493.33203125 disk.rbd0.out‚0' disk.nvme1c1n1.out‚5281.43994140625 cpu‚25.70374764418004 proc.cpu.threads‚17 network.sent‚ 2.961572e+06( disk./.usagePercent‚76.0730064452966 disk.md0.out‚ 53.0078125 disk.dm-0.in‚ 8.62109375" disk.dm-1.out‚2554.35888671875% disk.nvme0c0n1.out‚5607.244140625# proc.memory.rssMB‚ 2901.26171875) proc.memory.percent‚17.70789623260498$ disk./.usageGB‚1167.910831451416 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0! disk.dm-0.out‚8221.1396484375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0b w÷ž’ ‚(:“ ‘ðöÒð«†ß gpu.0.gpu‚90.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚271.101' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.36699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¯o“ ‚(:„  ðöÒéö‹Þ network.sent‚ 3.012921e+06 network.recv‚506388) disk./.usagePercent‚76.07220461966064 disk.nvme0c0n1.in‚ 7.1640625 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.md0.out‚53.3125 disk.rbd0.out‚0' disk.nvme1c1n1.out‚5392.74365234375% disk.nvme1n1.out‚5392.74365234375 disk.md0.in‚0# proc.memory.rssMB‚ 2901.26171875 cpu‚25.706281177459058% disk./.usageGB‚1167.8985214233398! disk.dm-0.out‚8646.5927734375& disk.nvme0c0n1.out‚5922.1416015625$ disk.nvme0n1.out‚5922.1416015625 disk.rbd0.in‚0) proc.memory.percent‚17.70789623260498 proc.cpu.threads‚17 disk.dm-0.in‚ 8.62109375 disk.dm-1.in‚0" disk.dm-1.out‚2554.42919921875 disk.nvme0n1.in‚ 7.1640625 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚96.99094295501709( proc.memory.availableMB‚ 493.003906258F“” ‚(:ˆ  ðöÒÉöÒÞ gpu.0.gpu‚72.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.557' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.519 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂΕ ‚(:‚ ¯ðöÒ£Ž¯Þ disk.md0.out‚ 53.4453125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 7.1640625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1c1n1.out‚7016.091796875# disk.nvme1n1.out‚7016.091796875$ memory_percent‚96.98505401611328 network.sent‚ 3.017029e+06 network.recv‚509821% disk./.usageGB‚1169.8646049499512 disk.md0.in‚0" disk.dm-0.out‚10614.5732421875 disk.nvme1n1.in‚ 1.45703125% proc.memory.availableMB‚ 493.96875! proc.memory.rssMB‚ 2900.296875) disk./.usagePercent‚76.20026737990237 disk.dm-0.in‚ 8.62109375" disk.dm-1.out‚2554.50732421875 disk.rbd1.in‚0' disk.nvme0c0n1.out‚6267.15771484375 disk.nvme0n1.in‚ 7.1640625% disk.nvme0n1.out‚6267.15771484375 disk.rbd0.in‚0 disk.rbd1.out‚0* proc.memory.percent‚17.702007293701172 cpu‚25.708887441639398 proc.cpu.threads‚17WƒÙ ž– ‚(:“ ¯ðöÒ‘ÒôÞ gpu.0.gpu‚66.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚209.153' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚69.71766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-?2— - _wandbruntime‚1870  _runtime‚1870UÚÿŸÉ˜ à trainer/global_step‚1999 % _timestamp‚1.7845269058490772e+09 _runtime‚1870.117241419 _step‚160 ' lr-AdamW/pg1‚0.00017586051264205987 ' lr-AdamW/pg2‚0.00017586051264205987äÍΙ È _runtime‚1870.117241419 _step‚160 ' lr-AdamW/pg1‚0.00017586051264205987 ' lr-AdamW/pg2‚0.00017586051264205987 trainer/global_step‚1999 % _timestamp‚1.7845269058490772e+09 ÙÜA2š - _wandbruntime‚1870  _runtime‚1870c„“ß'› ÙO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306658729901056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16081137 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68897474979328 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68897351097856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8248738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304353954838016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2318371585024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚161 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306659088364544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2304775063040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2304775063040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3490077 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16081135 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2305116487168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301623903582 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19719418 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66579084782080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304883188414420 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302581905072112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚1999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306658729901056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2305116487168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2318266315776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304353971877376 % _timestamp‚1.7845269058501604e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306659088364544 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2305116487168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2318266315776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304353971877376 % _timestamp‚1.7845269058501604e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306659088364544 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306658729901056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16081137 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68897474979328 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68897351097856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8248738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304353954838016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2318371585024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚161 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306659088364544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2304775063040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2304775063040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3490077 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16081135 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2305116487168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301623903582 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19719418 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66579084782080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304883188414420 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302581905072112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚1999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306658729901056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16088928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66612283664384 % _timestamp‚1.7845269064794843e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4760609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16088928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2319381222400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305031727470144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19729088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302729352778944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304502246379520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19718931 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4760907 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68931774343168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35807859 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306808472972800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19729088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚1999 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306808472972800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302732291946 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304502229340160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2305885161984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305032085070890 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2305885161984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2306226593280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8252420 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68931646274560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3491811 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2302391730560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488¯î&Ÿ èM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16088930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16088928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66612283664384 % _timestamp‚1.7845269064794843e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4760609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16088928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2319381222400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305031727470144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19729088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302729352778944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304502246379520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19718931 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4760907 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68931774343168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35807859 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306808472972800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19729088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚1999 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306808472972800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302732291946 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304502229340160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2305885161984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305032085070890 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2305885161984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2306226593280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8252420 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68931646274560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3491811 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2302391730560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2306226593280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66612265052160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302729335739584 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304502229340160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B¡ ÿ 0loss/FRt1/assignment_loss‚0.9911150276660919 1loss/FBt1/assignment_loss‚0.17530593401706154 /loss/FRt2/detection_loss‚0.5987027794762679 &loss/total_loss‚12.361246998915048 $ _timestamp‚1.784526906480602e+09 _runtime‚1870.748490379 2loss/SRqqt1/assignment_loss‚0.6507088880450896 2loss/SRqqt2/assignment_loss‚0.7855236269509737 1loss/SRqqt2/detection_loss‚0.9837766910337807 /loss/FBt1/detection_loss‚1.3241006337106391 epoch‚4 _step‚163 0loss/FRt2/assignment_loss‚2.5169971402823124 /loss/FRt1/detection_loss‚1.4077064212653427 /loss/FBt2/detection_loss‚1.9649831078676205 /loss/FBt2/assignment_loss‚0.255284165408196 1loss/SRqqt1/detection_loss‚0.7070425831916717 trainer/global_step‚1999—:ê‘ ¢ „ 0loss/FRt1/assignment_loss‚0.9911150276660919 1loss/FBt1/assignment_loss‚0.17530593401706154 /loss/FRt2/detection_loss‚0.5987027794762679 &loss/total_loss‚12.361246998915048 $ _timestamp‚1.784526906480602e+09 _runtime‚1870.748490379 2loss/SRqqt1/assignment_loss‚0.6507088880450896 2loss/SRqqt2/assignment_loss‚0.7855236269509737 1loss/SRqqt2/detection_loss‚0.9837766910337807 /loss/FBt1/detection_loss‚1.3241006337106391 epoch‚4 _step‚163 0loss/FRt2/assignment_loss‚2.5169971402823124 /loss/FRt1/detection_loss‚1.4077064212653427 /loss/FBt2/detection_loss‚1.9649831078676205 /loss/FBt2/assignment_loss‚0.255284165408196 1loss/SRqqt1/detection_loss‚0.7070425831916717 trainer/global_step‚1999£8&—’£ ‚(:‡ ¾ðöÒÌ°†Þ* proc.memory.percent‚17.702150344848633$ disk./.usageGB‚1170.626880645752 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚7019.85791015625 disk.md0.out‚ 54.16015625 disk.nvme0n1.in‚ 7.1640625 disk.rbd0.out‚0 disk.rbd1.out‚0 proc.cpu.threads‚17 network.recv‚513330) disk./.usagePercent‚76.24991894777736 disk.md0.in‚0 disk.dm-0.in‚ 8.62109375" disk.dm-1.out‚2554.55810546875$ disk.nvme0n1.out‚6283.7314453125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 7.1640625' disk.nvme1c1n1.out‚7019.85791015625$ memory_percent‚96.98688983917236( proc.memory.availableMB‚ 493.66796875 cpu‚25.716756875796417 network.sent‚ 3.019951e+06! disk.dm-0.out‚10633.361328125& disk.nvme0c0n1.out‚6283.7314453125 disk.rbd1.in‚0" proc.memory.rssMB‚ 2900.3203125XÓPž¤ ‚(:“ ¾ðöÒ÷ŸýÞ gpu.0.gpu‚63.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í·M<2¥ - _wandbruntime‚1887  _runtime‚1887M"æ. :¦ ¢{ =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.057176750153303146 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8054799437522888 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11072833091020584 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9902090430259705 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24505853652954102 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7270818948745728 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3545352518558502 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4583333134651184 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.828396737575531 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7208371162414551 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1905325949192047 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6564755439758301 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9661754369735718 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6722491979598999 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 jet/accuracy_5_of_5‚0 % _timestamp‚1.7845269236509488e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9844626188278198 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8293083906173706 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23658744990825653 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18305771052837372 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02496953494846821 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.06136767193675041 particle/accuracy_2_of_4‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2050960659980774 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29978007078170776 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19850905239582062 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_4_of_4‚0.009931244887411594 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7064275145530701 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7778630256652832 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9815350770950317 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48353949189186096 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4424712657928467 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6418367624282837 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7038691639900208 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9620814323425293 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5670772790908813 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24505853652954102 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7063803672790527 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4448869824409485 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.828396737575531 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8140588402748108 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03978148102760315 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9144726991653442 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9804443120956421 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_1_of_1‚0.32466864585876465 'particle/f_score‚0.6213797330856323 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961389899253845 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5900012254714966 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5092152953147888 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08359285444021225 +jet/accuracy_2_of_2‚0.13414610922336578 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9482865333557129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025635506957769394 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7493892312049866 +jet/accuracy_3_of_4‚0.08170447498559952 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4377967417240143 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9932966828346252 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9572363495826721 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38707324862480164 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17719222605228424 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *loss/val_total_loss‚11.606449663666295 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.0941864475607872 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 +particle/sensitivity‚0.6098532676696777 particle/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6142669320106506 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9542618989944458 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17198604345321655 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9821843504905701 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4075823426246643 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018744071945548058 *jet/accuracy_2_of_4‚0.7627815008163452 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4969339370727539 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7080012559890747 /particle/accuracy_4_of_4‚0.9582025408744812 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8612989783287048 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4004629850387573 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25179755687713623 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9847181439399719 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03643999248743057 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.276416540145874 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22729584574699402 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15783385932445526 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6018887758255005 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03385397419333458 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6791110634803772 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7782934904098511 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8404057025909424 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.985399603843689 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6084481477737427 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5861111879348755 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25399965047836304 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7893801927566528 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6023482084274292 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03926527127623558 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6340062618255615 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_3‚0.9050382971763611 *jet/accuracy_2_of_3‚0.5021122694015503 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10008011758327484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6536014080047607 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9787681102752686 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936005473136902 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_3_of_4‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8614113330841064 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7035465836524963 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6250823736190796 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5831412076950073 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932821989059448 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1947583705186844 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6060565114021301 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20857638120651245 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7817322611808777 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6738697290420532 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6060565114021301 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5686824917793274 +particle/specificity‚0.9034486413002014 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9950512647628784 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6073144674301147 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8643115758895874 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7036576271057129 *jet/accuracy_1_of_4‚0.9573726654052734 /particle/accuracy_3_of_3‚0.9995090961456299 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.47374647855758667 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35805392265319824 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 _step‚164 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31584084033966064 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 trainer/global_step‚1999 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5072005391120911 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04874974116683006 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018744071945548058 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5260416865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953699707984924 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.60435551404953 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24248811602592468 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8367155194282532 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6048194169998169 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9932966828346252 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9709656238555908 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9910759925842285 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029159244149923325 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2132505476474762 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.033571429550647736 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37601104378700256 Ó{úkŸ9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2525719702243805 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17342796921730042 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7310057878494263 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28167492151260376 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6497421860694885 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6481420993804932 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16415007412433624 7validation_average_jet_accuracy‚0.36046841740608215 epoch‚4 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6501569747924805 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16878704726696014 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _runtime‚1887.919648792¸(ë­=§ §{ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9847181439399719 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03643999248743057 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.276416540145874 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22729584574699402 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15783385932445526 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6018887758255005 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03385397419333458 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6791110634803772 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7782934904098511 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8404057025909424 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.985399603843689 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6084481477737427 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5861111879348755 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25399965047836304 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7893801927566528 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6023482084274292 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03926527127623558 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6340062618255615 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_3‚0.9050382971763611 *jet/accuracy_2_of_3‚0.5021122694015503 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10008011758327484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6536014080047607 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9787681102752686 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936005473136902 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_3_of_4‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8614113330841064 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7035465836524963 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6250823736190796 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5831412076950073 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932821989059448 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1947583705186844 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6060565114021301 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20857638120651245 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7817322611808777 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6738697290420532 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6060565114021301 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5686824917793274 +particle/specificity‚0.9034486413002014 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9950512647628784 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6073144674301147 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8643115758895874 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7036576271057129 *jet/accuracy_1_of_4‚0.9573726654052734 /particle/accuracy_3_of_3‚0.9995090961456299 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.47374647855758667 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35805392265319824 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 _step‚164 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31584084033966064 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 trainer/global_step‚1999 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5072005391120911 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04874974116683006 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018744071945548058 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5260416865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953699707984924 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.60435551404953 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24248811602592468 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8367155194282532 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6048194169998169 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9932966828346252 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9709656238555908 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9910759925842285 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029159244149923325 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2132505476474762 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.033571429550647736 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37601104378700256 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2525719702243805 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17342796921730042 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7310057878494263 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28167492151260376 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6497421860694885 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6481420993804932 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16415007412433624 7validation_average_jet_accuracy‚0.36046841740608215 epoch‚4 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6501569747924805 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16878704726696014 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _runtime‚1887.919648792 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.057176750153303146 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8054799437522888 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11072833091020584 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9902090430259705 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24505853652954102 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7270818948745728 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3545352518558502 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4583333134651184 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.828396737575531 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7208371162414551 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1905325949192047 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6564755439758301 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9661754369735718 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6722491979598999 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 jet/accuracy_5_of_5‚0 % _timestamp‚1.7845269236509488e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9844626188278198 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8293083906173706 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23658744990825653 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18305771052837372 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02496953494846821 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.06136767193675041 particle/accuracy_2_of_4‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2050960659980774 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29978007078170776 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19850905239582062 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_4_of_4‚0.009931244887411594 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7064275145530701 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7778630256652832 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9815350770950317 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48353949189186096 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4424712657928467 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6418367624282837 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7038691639900208 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9620814323425293 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5670772790908813 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24505853652954102 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7063803672790527 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4448869824409485 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.828396737575531 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8140588402748108 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03978148102760315 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9144726991653442 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9804443120956421 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_1_of_1‚0.32466864585876465 'particle/f_score‚0.6213797330856323 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961389899253845 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5900012254714966 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5092152953147888 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08359285444021225 +jet/accuracy_2_of_2‚0.13414610922336578 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9482865333557129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025635506957769394 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7493892312049866 +jet/accuracy_3_of_4‚0.08170447498559952 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4377967417240143 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9932966828346252 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9572363495826721 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38707324862480164 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17719222605228424 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *loss/val_total_loss‚11.606449663666295 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.0941864475607872 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 +particle/sensitivity‚0.6098532676696777 particle/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6142669320106506 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9542618989944458 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17198604345321655 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9821843504905701 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4075823426246643 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018744071945548058 *jet/accuracy_2_of_4‚0.7627815008163452 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4969339370727539 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7080012559890747 /particle/accuracy_4_of_4‚0.9582025408744812 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8612989783287048 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4004629850387573 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25179755687713623 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363¤¿”_¼„¨ ‚(:ù ÍðöÒ±ôÍÞ disk.nvme0n1.in‚ 7.1640625 network.sent‚ 3.09662e+06 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.62109375 disk.dm-1.out‚ 2800.0390625 cpu‚25.732739348716855% disk./.usageGB‚1170.6270141601562 disk.rbd0.in‚0" disk.nvme1n1.out‚ 7265.79296875 disk.md0.in‚0! disk.dm-0.out‚10634.009765625 disk.nvme0c0n1.in‚ 7.1640625$ memory_percent‚95.47405242919922# proc.memory.rssMB‚ 2899.91796875 disk.rbd1.in‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚6284.638671875% proc.memory.availableMB‚ 741.53125* proc.memory.percent‚17.699694633483887 proc.cpu.threads‚17 network.recv‚518121) disk./.usagePercent‚76.24992764436777 disk.nvme1c1n1.in‚ 1.45703125$ disk.nvme1c1n1.out‚ 7265.79296875 disk.rbd1.out‚0 disk.md0.out‚ 54.49609375% disk.nvme0c0n1.out‚6284.6386718753iž© ‚(:“ ÍðöÒ•¢ß gpu.0.gpu‚66.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚283.165' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.38833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µL{=‚ª ‚(:÷ ÜðöÒ®è¢Þ proc.cpu.threads‚17% disk.nvme0c0n1.out‚8932.349609375# disk.nvme0n1.out‚8932.349609375 disk.rbd1.in‚0" disk.nvme1n1.out‚ 8039.87109375 disk.md0.in‚0 disk.md0.out‚ 54.9609375! disk.dm-0.out‚13809.341796875 disk.dm-1.in‚0 network.sent‚ 3.141657e+06 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 disk.dm-0.in‚ 8.64453125 disk.dm-1.out‚ 3045.49609375( proc.memory.availableMB‚ 738.85546875# proc.memory.rssMB‚ 2902.58203125 network.recv‚522139) disk./.usagePercent‚76.42686822441188 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 8039.87109375* proc.memory.percent‚17.715954780578613 cpu‚25.730411364984814% disk./.usageGB‚1173.3434944152832 disk.nvme0n1.in‚7.1875 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.49038410186768¥©o;ž« ‚(:“ ÜðöÒÛàáÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.94133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚øú…¬ ‚(:ú ëðöÒÏÉìÝ( proc.memory.availableMB‚ 739.05078125 proc.cpu.threads‚17 network.sent‚ 3.14468e+06 network.recv‚525830 disk.md0.out‚ 55.3828125 disk.dm-0.in‚ 8.64453125% disk.nvme0c0n1.out‚8933.306640625 disk.nvme0n1.in‚7.1875# disk.nvme0n1.out‚8933.306640625 disk.rbd0.in‚0) disk./.usagePercent‚76.41136990640203 disk.md0.in‚0 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0$ disk.nvme1n1.out‚8040.5380859375$ memory_percent‚95.48919200897217$ disk./.usageGB‚1173.105556488037 disk.dm-1.in‚0! disk.dm-1.out‚3045.5810546875# proc.memory.rssMB‚ 2902.58203125* proc.memory.percent‚17.715954780578613 cpu‚25.72838008000259! disk.dm-0.out‚13809.974609375& disk.nvme1c1n1.out‚8040.5380859375 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125é ­ ‚(:’ ëðöÒ­´ÂÞ gpu.0.gpu‚45.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚279.25' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.08333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^íaJ2® - _wandbruntime‚1934  _runtime‚1934S'\%ǯ Á % _timestamp‚1.7845269697722137e+09 _runtime‚1934.040446736 _step‚165 & lr-AdamW/pg1‚0.0001756761313849034 & lr-AdamW/pg2‚0.0001756761313849034 trainer/global_step‚2049qF]Ì° Æ & lr-AdamW/pg1‚0.0001756761313849034 & lr-AdamW/pg2‚0.0001756761313849034 trainer/global_step‚2049 % _timestamp‚1.7845269697722137e+09 _runtime‚1934.040446736 _step‚165¥ÆSi2± - _wandbruntime‚1934  _runtime‚1934iÂh_Þ'² ØO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2381003427328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316090685824000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20285569 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316091044287488 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70655748514304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16593595 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377070404618 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20275426 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313710023820800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16593595 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2049 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36869019 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16593593 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316090685824000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20275426 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2381003427328 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8492719 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3580893 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70655630924288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311874694453936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313710040860160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ¿j³ ÝO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2394953975808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314251764858554 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311874711493296 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2377410965892 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36879164 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316091044287488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20285569 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_stajìÃÊ"rt/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316090685824000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20285569 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316091044287488 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70655748514304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16593595 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377070404618 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20275426 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313710023820800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16593595 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2049 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36869019 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16593593 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316090685824000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20275426 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2381003427328 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8492719 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3580893 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70655630924288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311874694453936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313710040860160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2381003427328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312022125121408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8496682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70689921128448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚1934.672364414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313858298322944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2396167074816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16601388 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68293875830784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2396063909888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68293857218560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚167 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16601388 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4914055 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316240428895744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2382113533440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2381772102144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312022125121408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8496682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70689921128448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚1934.672364414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313858298322944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2396167074816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16601388 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68293875830784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2396063909888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68293857218560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚167 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16601388 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4914055 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316240428895744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2382113533440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2381772102144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3667596 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2450941434368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16983245 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16983243 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20760078 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321679139046196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚2099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16983243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323559980033024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2450842456576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37743321 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319246227827536 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8694848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321123748928000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319246244866896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚1979.549998923 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5026984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2436572529152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323560338496512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321123765967360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3667593 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321678781445562 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319246227827536 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8694848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321123748928000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319246244866896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚1979.549998923 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5026984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2436572529152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323560338496512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321123765967360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3667593 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321678781445562 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3667596 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2450941434368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16983245 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16983243 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20760078 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321679139046196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚2099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16983243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323559980033024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2450842456576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37743321 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8698562 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20779891 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72404827806720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321272023430144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319393675534368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20769734 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321272040469504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16991038 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323709723104768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚2099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2437682635264 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16991036 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321272023430144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20769734 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72404945389568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321272040469504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321827320501286 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319393658495008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845270159158313e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20779891 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37760770 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16991038»–ží&Ê çM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845270159158313e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20779891 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37760770 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16991038 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37760770 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2452056054784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3669330 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323709364634112 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8698562 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20779891 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72404827806720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321272023430144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319393675534368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20769734 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321272040469504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16991038 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323709723104768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚2099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2437682635264 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16991036 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321272023430144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20769734 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72404945389568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321272040469504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321827320501286 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319393658495008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331029274242048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328537474035200 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚2024.831476077 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2492141630976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5141426 trainer/global_step‚2149 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328537474035200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2488377392708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74085541281280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2506818712576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17372895 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331029274242048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328537491074560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2491800206848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3754296 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74085417399808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71578722568704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17372895 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71578703956480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2506713443328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845270605632598e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329106155633204 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17372893 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331029632705536 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5141709 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21254873 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3754293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8896005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚174 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328537491074560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329105798032570 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2488036831434 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17372893 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21254873 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180B-( 5× ÞO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74085541281280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2506818712576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17372895 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331029274242048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328537491074560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2491800206848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3754296 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74085417399808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71578722568704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17372895 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71578703956480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2506713443328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845270605632598e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329106155633204 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17372893 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331029632705536 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5141709 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21254873 SGDeviceStIû¯¯atsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3754293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8896005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚174 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328537491074560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329105798032570 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2488036831434 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17372893 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21254873 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331029274242048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328537474035200 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚2024.831476077 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2492141630976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5141426 trainer/global_step‚2149 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328537474035200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2488377392708®¼}" ‚Ø ‚(:÷ ÔñöÒãóÞ disk.nvme1n1.in‚ 1.45703125# memory_percent‚95.4944372177124# proc.memory.rssMB‚ 2902.58203125* proc.memory.percent‚17.715954780578613) disk./.usagePercent‚76.24683538529136 disk.rbd1.out‚0" disk.dm-0.out‚14949.3056640625 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 cpu‚25.71576872719589 network.sent‚ 3.330272e+06 disk.rbd0.in‚0 disk.rbd1.in‚0! disk.nvme1n1.out‚ 8348.6328125 disk.dm-0.in‚ 8.64453125 disk.dm-1.out‚ 3046.28515625 disk.nvme0c0n1.in‚7.1875 proc.cpu.threads‚17 network.recv‚551973% disk./.usageGB‚1170.5795402526855 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.in‚0 disk.md0.out‚ 57.88671875& disk.nvme0c0n1.out‚9770.6533203125$ disk.nvme0n1.out‚9770.6533203125( proc.memory.availableMB‚ 738.19140625 disk.rbd0.out‚0# disk.nvme1c1n1.out‚ 8348.6328125ÔPržÙ ‚(:“ ÔñöÒåÕ…ß gpu.0.gpu‚71.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚134.189' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚44.72966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³62Ú - _wandbruntime‚2025  _runtime‚2025Há5Ôé&Û ãM _step‚175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17380686 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2493251737088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8899747 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5143720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328685748537344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331178658843136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2492910305792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74119706620928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71611902838784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326765208907968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71611884226560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21254386 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21264543 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17380688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3756027 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329254694689040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17380688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38635072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326765191868608 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2507822394368 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8900041 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21254386 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 % _timestamp‚1.7845270615374353e+09 _runtime‚2025.805756183 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74119838883840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17380686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38635072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2493251737088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2489145219686 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38645231 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21264543 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2492910305792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2489145219686 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38645231 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21264543 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2492910305792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2493251737088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8899747 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5143720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328685748537344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331178658843136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2492910305792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74119706620928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71611902838784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326765208907968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71611884226560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21254386 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21264543 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17380688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3756027 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329254694689040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17380688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38635072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326765191868608 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2507822394368 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8900041 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21254386 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 % _timestamp‚1.7845270615374353e+09 _runtime‚2025.805756183 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74119838883840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17380686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38635072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2493251737088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296¯HÕn,2Ý - _wandbruntime‚2025  _runtime‚2025zÞ ÿ _step‚176 2loss/SRqqt1/assignment_loss‚0.6413019159243512 1loss/FBt1/assignment_loss‚0.09114946891213564 2loss/FBt2/assignment_loss‚0.061575356780405766 1loss/SRqqt1/detection_loss‚0.7220071919359511 -loss/FBt2/detection_loss‚2.62511982969781 0loss/FRt1/assignment_loss‚1.0779930906466755 2loss/SRqqt2/assignment_loss‚0.6515104383044088 /loss/FBt1/detection_loss‚0.7291116368273489 epoch‚5 trainer/global_step‚2149 % _timestamp‚1.7845270615385735e+09 /loss/FRt2/detection_loss‚0.5593062334606385 0loss/SRqqt2/detection_loss‚1.007168700219704 &loss/total_loss‚12.007028483603825 _runtime‚2025.806416914 /loss/FRt2/assignment_loss‚2.366775583317223 /loss/FRt1/detection_loss‚1.4740090375771724ºhòâ ß „ epoch‚5 trainer/global_step‚2149 % _timestamp‚1.7845270615385735e+09 /loss/FRt2/detection_loss‚0.5593062334606385 0loss/SRqqt2/detection_loss‚1.007168700219704 &loss/total_loss‚12.007028483603825 _runtime‚2025.806416914 /loss/FRt2/assignment_loss‚2.366775583317223 /loss/FRt1/detection_loss‚1.4740090375771724 _step‚176 2loss/SRqqt1/assignment_loss‚0.6413019159243512 1loss/FBt1/assignment_loss‚0.09114946891213564 2loss/FBt2/assignment_loss‚0.061575356780405766 1loss/SRqqt1/detection_loss‚0.7220071919359511 -loss/FBt2/detection_loss‚2.62511982969781 0loss/FRt1/assignment_loss‚1.0779930906466755 2loss/SRqqt2/assignment_loss‚0.6515104383044088 /loss/FBt1/detection_loss‚0.7291116368273489°½‡†‚à ‚(:÷ ãñöÒÌ»ÏÞ network.recv‚556418 disk.nvme1n1.in‚ 1.45703125# disk.nvme1n1.out‚8352.248046875 disk.md0.out‚ 58.015625" disk.dm-0.out‚14973.5791015625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1c1n1.out‚8352.248046875 network.sent‚ 3.3948e+06 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0# memory_percent‚95.4880952835083# proc.memory.rssMB‚ 2901.69140625* proc.memory.percent‚17.710518836975098$ disk./.usageGB‚1170.580478668213 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚9794.955078125# disk.nvme0n1.out‚9794.955078125( proc.memory.availableMB‚ 739.23046875) disk./.usagePercent‚76.24689650989825! disk.dm-1.out‚3049.6220703125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 cpu‚25.713573754035384 proc.cpu.threads‚17bûdá ‚(:’ ãñöÒœÌ–ß gpu.0.gpu‚30.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.304' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.10133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B¨=°â ‚(:ô òñöÒ±³šÞ# disk.nvme0n1.out‚11710.64453125 disk.rbd0.in‚0 disk.nvme1n1.out‚ 8353.078125" disk.dm-0.out‚16888.9541015625 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125" disk.nvme1c1n1.out‚ 8353.078125 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125! disk.dm-1.out‚3049.7236328125 disk.nvme0c0n1.in‚7.1875$ disk./.usageGB‚1173.341625213623 disk.md0.out‚ 58.50390625( proc.memory.availableMB‚ 738.14453125* proc.memory.percent‚17.715954780578613 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0% disk.nvme0c0n1.out‚11710.64453125$ memory_percent‚95.49472332000732# proc.memory.rssMB‚ 2902.58203125 cpu‚25.71167583846823 network.sent‚ 3.435838e+06 network.recv‚560070( disk./.usagePercent‚76.4267464721461,Ëã ‚(:’ òñöÒµÚäÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.758' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.919333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â«Ðä ‚(:… òöÒ¦é“Þ$ memory_percent‚95.49322128295898* proc.memory.percent‚17.715954780578613 cpu‚25.709915626523692 disk.rbd0.in‚0 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚11711.52783203125 disk.nvme0n1.in‚7.1875 disk.nvme1n1.in‚ 1.45703125 proc.cpu.threads‚17 network.sent‚ 3.440833e+06 network.recv‚563861" disk.dm-0.out‚16889.5361328125& disk.nvme0n1.out‚11711.52783203125 disk.md0.out‚ 58.87109375& proc.memory.availableMB‚ 738.390625) disk./.usagePercent‚76.42675641110658% disk./.usageGB‚1173.3417778015137 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0! disk.dm-1.out‚3050.0029296875 disk.rbd1.in‚0 disk.rbd1.out‚0# proc.memory.rssMB‚ 2902.58203125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚8353.85400390625% disk.nvme1n1.out‚8353.85400390625 disk.md0.in‚0Ü+9¤å ‚(:’ òöÒ‚àÔÞ gpu.0.gpu‚84.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚211.87' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.62333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú2æ - _wandbruntime‚2070  _runtime‚2070£»­ïÇç Á _runtime‚2070.364751053 _step‚177 & lr-AdamW/pg1‚0.0001750897848570903 & lr-AdamW/pg2‚0.0001750897848570903 trainer/global_step‚2199 % _timestamp‚1.7845271060965416e+09†ïžûÌè Æ _runtime‚2070.364751053 _step‚177 & lr-AdamW/pg1‚0.0001750897848570903 & lr-AdamW/pg2‚0.0001750897848570903 trainer/global_step‚2199 % _timestamp‚1.7845271060965416e+09±ñV˜¥2é - _wandbruntime‚2070  _runtime‚2070çÈÍüê ÙO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3840996 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39491925 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9097755 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17762543 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2562676167680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2543520044842 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_bþuÕã!atch_start/active_bytes.small_pool.freed‚ 2547369308672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2547369308672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3840993 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333989311614096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚2199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2547710732800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21729382 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2543860606116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚335951199142400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21729382 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39502070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338498926914560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2070.365950325 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17762545 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336532814619578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21739525 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335951199142400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21739525 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9097461 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75800288358912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336533172220212 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338498568451072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5256468 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75800412240384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338498568451072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338498926914560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335951216181760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17762545 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333989294574736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184¨rÞä'ë ÞO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2547369308672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3840993 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333989311614096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚2199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2547710732800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21729382 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2543860606116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚335951199142400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21729382 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39502070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338498926914560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2070.365950325 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17762545 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336532814619578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21739525 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335951199142400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21739525 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9097461 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75800288358912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336533172220212 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338498568451072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5256468 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75800412240384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338498568451072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338498926914560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335951216181760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17762545 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333989294574736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3840996 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39491925 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9097755 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17762543 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2562676167680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2543520044842 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2547369308672²mÏ3z2ì - _wandbruntime‚2070  _runtime‚2070Ôü´™é&í ãM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334136742281568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39509374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73270897730560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336681711276048 _step‚179 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3842727 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334136725242208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17770336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336099490683904 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336099473644544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336099473644544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚2199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845271067309158e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2548820838912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9101715 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338648311522816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21749195 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17770338 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2563796221440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21749195 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9101413 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39519533 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17770336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2070.999257455 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2548479407616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5258686 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21739038 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2548479407616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2544968994480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338648311522816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3366813536753029Àÿûî èM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17770338 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2544628433094 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338647953052160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2548479407616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5258686 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21739038 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2548479407616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2544968994480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338648311522816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336681353675302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334136742281568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39509374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73270897730560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336681711276048 _step‚179 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3842727 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334136725242208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17770336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336099490683904 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336099473644544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336099473644544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚2199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845271067309158e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2548820838912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9101715 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338648311522816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21749195 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17770338 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2563796221440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21749195 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9101413 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39519533 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17770336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2070.999257455 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 $ _timestamp‚1.784527151862856e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2618505481216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5370904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3927696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345967862660096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40366227 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345967862660096 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343959831206586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343364941288960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2618402309120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77515133273600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2602938410496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343960188807220 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2603279834624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74896749576704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341360844987696 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2599343819524 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18152195 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚182 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5371180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2116.131060701 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180¿ ðã'ý ÝO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343364941288960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2618402309120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77515133273600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2602938410496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343960188807220 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2603279834624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74896749576704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341360844987696 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2599343819524 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18152195 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚182 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5371180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2116.131060701 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2602938410496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18152195 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343364924249600 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9298876 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 $ _timestamp‚1.784527151862856e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2618505481216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5370904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3927696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345967862660096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40366227 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345967862660096 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343959831206586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¶[HÕH"þ - _wandbruntime‚2116   K_runtime‚2116Ø Yé&ÿ ãM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2619517375488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18159986 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚343513198751744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18159988 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40393835 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22233847 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343513198751744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22233847 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74929911234560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2619620540416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346117605731840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341508258615808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2604389940736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3929430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344108370262310 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74929929846784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9302573 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341508275655168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2600452207888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22223690 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2604389940736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5373427 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40383676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9302857 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 trainer/global_step‚2249“¥àî&€ èM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22223690 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2604389940736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5373427 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40383676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9302857 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 trainer/global_step‚2249 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2619517375488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18159986 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚343513198751744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18159988 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40393835 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22233847 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343513198751744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22233847 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74929911234560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2619620540416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346117605731840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341508258615808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2604389940736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3929430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344108370262310 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74929929846784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9302573 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341508275655168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2600452207888·è†a2 - _wandbruntime‚2116  _runtime‚2116‚àæ‚ ‚ 2loss/SRqqt1/assignment_loss‚0.5658286274787594 2loss/SRqqt2/assignment_loss‚0.7130677581024227 1loss/SRqqt2/detection_loss‚1.1605493081086833 _step‚184 1loss/FBt1/assignment_loss‚0.14406400575533904 _runtime‚2116.763723297 1loss/SRqqt1/detection_loss‚0.6949318818316977 /loss/FBt2/detection_loss‚2.1087144625683627 &loss/total_loss‚11.617485331012963 0loss/FRt1/assignment_loss‚1.0004448947915132 2loss/FBt2/assignment_loss‚0.002625488941747436 /loss/FRt1/detection_loss‚1.3939599465836934 /loss/FRt2/detection_loss‚0.5825093831060797 /loss/FBt1/detection_loss‚1.0264112205438605 epoch‚5 trainer/global_step‚2249 % _timestamp‚1.7845271524957867e+09 /loss/FRt2/assignment_loss‚2.224378353200804ñTµQ ƒ ‡ 1loss/FBt1/assignment_loss‚0.14406400575533904 _runtime‚2116.763723297 1loss/SRqqt1/detection_loss‚0.6949318818316977 /loss/FBt2/detection_loss‚2.1087144625683627 &loss/total_loss‚11.617485331012963 0loss/FRt1/assignment_loss‚1.0004448947915132 2loss/FBt2/assignment_loss‚0.002625488941747436 /loss/FRt1/detection_loss‚1.3939599465836934 /loss/FRt2/detection_loss‚0.5825093831060797 /loss/FBt1/detection_loss‚1.0264112205438605 epoch‚5 trainer/global_step‚2249 % _timestamp‚1.7845271524957867e+09 /loss/FRt2/assignment_loss‚2.224378353200804 2loss/SRqqt1/assignment_loss‚0.5658286274787594 2loss/SRqqt2/assignment_loss‚0.7130677581024227 1loss/SRqqt2/detection_loss‚1.1605493081086833 _step‚184¸…Ó¿Œ„ ‚(: ½òöÒòî®Þ network.sent‚ 3.528094e+06 disk.md0.out‚ 60.12109375 disk.dm-0.in‚ 8.64453125& disk.nvme0n1.out‚12468.29248046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125$ disk.nvme1c1n1.out‚ 8486.38671875! disk.dm-1.out‚3050.4580078125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125* proc.memory.percent‚17.710518836975098 disk.dm-1.in‚0 disk.rbd1.out‚0" disk.nvme1n1.out‚ 8486.38671875$ memory_percent‚95.48256397247314( proc.memory.availableMB‚ 740.13671875# proc.memory.rssMB‚ 2901.69140625 cpu‚25.702644615415874 proc.cpu.threads‚17 network.recv‚577494) disk./.usagePercent‚76.42375484504444$ disk./.usageGB‚1173.295696258545 disk.md0.in‚0# disk.dm-0.out‚17775.67626953125 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚12468.29248046875 disk.nvme0n1.in‚7.1875‡ìí… ‚(:’ ½òöÒèÛúÞ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚253.12' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.37333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ìE)† ‚(:’ ÌòöÒßͶÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚280.91' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.63666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0szÚŒ‡ ‚(: ÌòöÒŠ†ºà disk.md0.out‚ 60.61328125* proc.memory.percent‚17.710518836975098 cpu‚25.700839558643835 proc.cpu.threads‚17 network.recv‚581543( disk./.usagePercent‚76.4237749714394 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚8487.3173828125 disk.rbd1.out‚0 disk.md0.in‚0 network.sent‚ 3.579471e+06" disk.dm-0.out‚17777.1728515625 disk.dm-1.in‚0! disk.dm-1.out‚3050.4970703125 disk.nvme0n1.in‚7.1875% disk.nvme0n1.out‚12469.9384765625$ memory_percent‚96.53468132019043' proc.memory.availableMB‚ 567.7578125 disk.dm-0.in‚ 8.64453125' disk.nvme0c0n1.out‚12469.9384765625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2901.69140625% disk./.usageGB‚1173.2960052490234 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125$ disk.nvme1n1.out‚8487.3173828125¹O?Žˆ ‚(:ƒ ÛòöÒÌÙƒÞ disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125# proc.memory.rssMB‚ 2902.58203125 disk.md0.out‚ 60.83203125 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚17777.8525390625 disk.rbd0.out‚0' disk.nvme1c1n1.out‚8487.73876953125 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚8487.73876953125$ memory_percent‚96.99079990386963) disk./.usagePercent‚76.42377124432923% disk./.usageGB‚1173.2959480285645 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0* proc.memory.percent‚17.715954780578613 cpu‚25.69928523162309" disk.dm-1.out‚3050.56201171875 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚12470.740234375( proc.memory.availableMB‚ 493.02734375 proc.cpu.threads‚17 network.sent‚ 3.582393e+06 network.recv‚585160 disk.nvme0c0n1.in‚7.1875& disk.nvme0c0n1.out‚12470.740234375®¤ó7‰ ‚(:’ ÛòöÒÝœÔÞ gpu.0.gpu‚11.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.94133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-¿™µ2Š - _wandbruntime‚2161  _runtime‚2161åxâ(É‹ à ' lr-AdamW/pg1‚0.00017467134462714778 ' lr-AdamW/pg2‚0.00017467134462714778 trainer/global_step‚2299 % _timestamp‚1.7845271971939974e+09 _runtime‚2161.462115092 _step‚1850£Ñ,ÎŒ È _step‚185 ' lr-AdamW/pg1‚0.00017467134462714778 ' lr-AdamW/pg2‚0.00017467134462714778 trainer/global_step‚2299 % _timestamp‚1.7845271971939974e+09 _runtime‚2161.462115092¹Ö~ó¯2 - _wandbruntime‚2161  _runtime‚2161¬:㎠ÙO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4014393 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚186 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18541845 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350778666396160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18541845 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5486444 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76555744468480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚2161.463088583 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2658848936448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79230101085696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79229977204224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353437156869120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674338004992 % _timestamp‚1.7845271971950123e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41250674 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353437515332608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2654827032932 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350778649356800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9500840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351387205394228 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚2299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353437515332608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2658848936448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351386847793594 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350778666396160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18541845 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5486444 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76555744468480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚2161.463088583 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2658848936448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79230101085696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79229977204224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353437156869120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674338004992 % _timestamp‚1.7845271971950123e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41250674 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353437515332608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2654827032932 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350778649356800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9500840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351387205394228 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚2299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353437515332608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2658848936448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351386847793594 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41268137 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22718499 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18549638 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚350926923858944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5488737 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 _runtime‚2162.093029993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22708342 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79264272427008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2659959042560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41268137 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353586899940864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353586899940864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4016127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675347688448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41257978 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18549636 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18549636 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2659617611264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76588943350784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350926940898304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353586541470208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41257978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9504867 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5488447 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9504574 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79264400495616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4016130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2675457144832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22718499 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350926923858944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2655935421296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845271978246393e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336z¤?Qâ"’ èM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4016130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2675457144832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22718499 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350926923858944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2655935421296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845271978246393e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41268137 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22718499 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18549638 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚350926923858944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5488737 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 _runtime‚2162.093029993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22708342 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79264272427008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2659959042560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41268137 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353586899940864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353586899940864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4016127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675347688448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41257978 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18549636 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18549636 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2659617611264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76588943350784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350926940898304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353586541470208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41257978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9504867 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5488447 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9504574 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79264400495616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488»-=952“ - _wandbruntime‚2162  _runtime‚2162~y ” € 2loss/FBt2/assignment_loss‚0.025905110544036394 /loss/FRt1/detection_loss‚1.3871996349545335 1loss/SRqqt2/detection_loss‚0.9719852076820872 /loss/FBt1/detection_loss‚0.6598652463069664 &loss/total_loss‚10.247734776107261 # _timestamp‚1.78452719782586e+09 _step‚188 /loss/FRt1/assignment_loss‚1.018458593408972 2loss/SRqqt2/assignment_loss‚0.6188922898950804 /loss/FRt2/detection_loss‚0.5644611357318735 /loss/FBt2/detection_loss‚1.4311218660685323 1loss/FBt1/assignment_loss‚0.08893444771701611 1loss/SRqqt1/detection_loss‚0.6404741053691976 _runtime‚2162.094141115 0loss/FRt2/assignment_loss‚2.2308163278389976 2loss/SRqqt1/assignment_loss‚0.6096208105899691 epoch‚5 trainer/global_step‚2299. Á • … 0loss/FRt2/assignment_loss‚2.2308163278389976 2loss/SRqqt1/assignment_loss‚0.6096208105899691 epoch‚5 trainer/global_step‚2299 2loss/FBt2/assignment_loss‚0.025905110544036394 /loss/FRt1/detection_loss‚1.3871996349545335 1loss/SRqqt2/detection_loss‚0.9719852076820872 /loss/FBt1/detection_loss‚0.6598652463069664 &loss/total_loss‚10.247734776107261 # _timestamp‚1.78452719782586e+09 _step‚188 /loss/FRt1/assignment_loss‚1.018458593408972 2loss/SRqqt2/assignment_loss‚0.6188922898950804 /loss/FRt2/detection_loss‚0.5644611357318735 /loss/FBt2/detection_loss‚1.4311218660685323 1loss/FBt1/assignment_loss‚0.08893444771701611 1loss/SRqqt1/detection_loss‚0.6404741053691976 _runtime‚2162.094141115¼ZUÇ– ‚(:‚ êòöÒð¸ÝÞ# disk.nvme1n1.out‚8488.576171875# proc.memory.rssMB‚ 2901.69140625 cpu‚25.697219381786237) disk./.usagePercent‚76.42378292260777$ disk./.usageGB‚1173.296127319336 disk.nvme0n1.in‚7.1875% disk.nvme1c1n1.out‚8488.576171875 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125# memory_percent‚96.9857931137085 network.recv‚588510" disk.dm-0.out‚17780.1279296875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚12473.14208984375( proc.memory.availableMB‚ 493.84765625* proc.memory.percent‚17.710518836975098 disk.md0.in‚0 disk.md0.out‚ 61.23046875" disk.dm-1.out‚3050.67138671875& disk.nvme0n1.out‚12473.14208984375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 3.600134e+06 disk.dm-0.in‚ 8.64453125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125S 𶞗 ‚(:“ êòöÒ¢é ß gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚257.492' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.83066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¬\†˜ ‚(:û ùòöÒ‚úüÝ& disk.nvme1c1n1.out‚8489.0771484375$ disk.nvme1n1.out‚8489.0771484375 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚17780.7412109375# proc.memory.rssMB‚ 2902.58203125* proc.memory.percent‚17.715954780578613 cpu‚25.695330474602113 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.out‚ 61.5234375 disk.dm-1.out‚3050.744140625) disk./.usagePercent‚76.42377422601737% disk./.usageGB‚1173.2959938049316 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.in‚0& proc.memory.availableMB‚ 492.953125 proc.cpu.threads‚17 network.sent‚ 3.649576e+06 network.recv‚592373% disk.nvme0c0n1.out‚12473.98828125 disk.nvme0n1.in‚7.1875# disk.nvme0n1.out‚12473.98828125 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0$ memory_percent‚96.99125289916992õrŽ›™ ‚(: ùòöÒüßÐÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.66' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.88666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~‘}0~š ‚(:ó ˆóöÒÞÉŸÞ disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0 disk.dm-1.out‚3050.787109375 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875$ memory_percent‚96.98584079742432" disk.nvme1n1.out‚ 8489.73046875$ disk.nvme0n1.out‚12475.591796875 disk.rbd0.out‚0( proc.memory.availableMB‚ 493.83984375* proc.memory.percent‚17.710518836975098 proc.cpu.threads‚17 network.sent‚ 3.655573e+06 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚ 17782.1875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2901.69140625 cpu‚25.69332965666794 network.recv‚595532( disk./.usagePercent‚76.4237814317637 disk.nvme1c1n1.in‚ 1.45703125$ disk.nvme1c1n1.out‚ 8489.73046875 disk.rbd1.out‚0 disk.md0.in‚0 disk.md0.out‚ 61.8828125& disk.nvme0c0n1.out‚12475.591796875% disk./.usageGB‚1173.2961044311523'¹¶ü“› ‚(:ˆ ˆóöÒîöÞ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚201.588' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 N2œ - _wandbruntime‚2207  _runtime‚2207œJŒÙÉ Ã % _timestamp‚1.7845272430336049e+09 _runtime‚2207.301755592 _step‚189 ' lr-AdamW/pg1‚0.00017445390153216886 ' lr-AdamW/pg2‚0.00017445390153216886 trainer/global_step‚2349® ÁΞ È ' lr-AdamW/pg1‚0.00017445390153216886 ' lr-AdamW/pg2‚0.00017445390153216886 trainer/global_step‚2349 % _timestamp‚1.7845272430336049e+09 _runtime‚2207.301755592 _step‚189½us”2Ÿ - _wandbruntime‚2207  _runtime‚2207–ù«ûß'  ÙO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2730059083776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358192391503360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18931495 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18931493 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚2207.305626638 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18931495 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2714076614144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚2349 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80944817056256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360906451078144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4101096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4101093 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9702955 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356103894695536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2710310246340 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78214776584704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5601859 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845272430344296e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2714076614144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23183338 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358814221981236n ò§ä'¡ ÞO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78214776584704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5601859 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845272430344296e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2714076614144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23183338 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358814221981236 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2730059083776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358192391503360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18931495 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18931493 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚2207.305626638 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18931495 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2714076614144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚2349 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80944817056256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360906451078144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4101096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4101093 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9702955 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356103894695536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2710310246340 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272¾1åi2¢ - _wandbruntime‚2207  _runtime‚2207Ù³êæ£ ãM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9706652 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5604109 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80979239519744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361056194149888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18939288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356251342402368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42142439 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2715186713088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2715186713088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358340666005504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356251325363008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358340648966144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2711418634704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78247938242560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42132280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _runtime‚2207.942177286 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5603825 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23203151 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚191 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358340666005504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18939286 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845272436731136e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18939286 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23192994 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42142439 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23203151 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4102827 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80979117742592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42132280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23192994 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731282664960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731179500032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚361055835679232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358962403436326ïííqî&¤ èM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358340666005504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18939286 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845272436731136e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18939286 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23192994 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42142439 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23203151 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4102827 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80979117742592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42132280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23192994 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731282664960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731179500032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚361055835679232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358962403436326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9706652 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5604109 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80979239519744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361056194149888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18939288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356251342402368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42142439 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2715186713088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2715186713088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358340666005504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356251325363008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358340648966144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2711418634704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78247938242560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42132280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _runtime‚2207.942177286 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5603825 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23203151 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚191 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¿Ê$2¥ - _wandbruntime‚2207  _runtime‚2207å_žÅ¦ ‚ 0loss/FRt2/assignment_loss‚2.2027993766368663 2loss/SRqqt1/assignment_loss‚0.5828382173812601 /loss/FBt2/detection_loss‚1.9921031749807299 _step‚192 %loss/total_loss‚11.10142859888111 epoch‚5 0loss/FRt1/assignment_loss‚0.9967888142772301 1loss/FBt1/assignment_loss‚0.09941759444441771 2loss/FBt2/assignment_loss‚0.005190100110438323 /loss/FRt2/detection_loss‚0.5861474478053887 1loss/SRqqt1/detection_loss‚0.7119600728268898 /loss/FBt1/detection_loss‚0.8001532659031133 2loss/SRqqt2/assignment_loss‚0.6324545549642294 trainer/global_step‚2349 _runtime‚2207.942997187 /loss/FRt1/detection_loss‚1.5080860703279675 1loss/SRqqt2/detection_loss‚0.9834899092225777 % _timestamp‚1.7845272436751003e+09;9)o § ‡ /loss/FBt2/detection_loss‚1.9921031749807299 _step‚192 %loss/total_loss‚11.10142859888111 epoch‚5 0loss/FRt1/assignment_loss‚0.9967888142772301 1loss/FBt1/assignment_loss‚0.09941759444441771 2loss/FBt2/assignment_loss‚0.005190100110438323 /loss/FRt2/detection_loss‚0.5861474478053887 1loss/SRqqt1/detection_loss‚0.7119600728268898 /loss/FBt1/detection_loss‚0.8001532659031133 2loss/SRqqt2/assignment_loss‚0.6324545549642294 trainer/global_step‚2349 _runtime‚2207.942997187 /loss/FRt1/detection_loss‚1.5080860703279675 1loss/SRqqt2/detection_loss‚0.9834899092225777 % _timestamp‚1.7845272436751003e+09 0loss/FRt2/assignment_loss‚2.2027993766368663 2loss/SRqqt1/assignment_loss‚0.5828382173812601ÀÁN¿«¨ ‚(:‚ —óöÒ´–œÞ disk.md0.out‚ 62.10546875# memory_percent‚96.9930648803711 network.sent‚ 3.6594e+06" disk.dm-1.out‚3050.92529296875 disk.rbd1.out‚0% disk.nvme1n1.out‚8490.24853515625* proc.memory.percent‚17.715954780578613 cpu‚25.691251087731136 proc.cpu.threads‚17 network.recv‚599422) disk./.usagePercent‚76.42379509783436 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125% proc.memory.availableMB‚ 492.65625# proc.memory.rssMB‚ 2902.58203125$ disk./.usageGB‚1173.296314239502 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚12476.78662109375 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚12476.78662109375' disk.nvme1c1n1.out‚8490.24853515625 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚17783.26806640625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0ÓâÕý“© ‚(:ˆ —óöÒð¡®Þ gpu.0.gpu‚56.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.897' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.299 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:#Õ|Žª ‚(:ƒ ¦óöÒ祜Þ& disk.nvme0n1.out‚12478.91650390625 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125& proc.memory.availableMB‚ 492.359375* proc.memory.percent‚17.715954780578613 network.sent‚ 3.723734e+06 network.recv‚603814 disk.rbd0.in‚0 disk.rbd1.out‚0% disk./.usageGB‚1173.2961349487305 disk.md0.out‚ 62.42578125 disk.nvme0c0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125& disk.nvme1c1n1.out‚8490.9169921875$ disk.nvme1n1.out‚8490.9169921875 cpu‚25.689532579727626 proc.cpu.threads‚17( disk./.usagePercent‚76.4237834195558 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚3050.98388671875( disk.nvme0c0n1.out‚12478.91650390625 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0$ memory_percent‚96.99487686157227# proc.memory.rssMB‚ 2902.58203125 disk.md0.in‚0" disk.dm-0.out‚17785.3115234375 disk.dm-1.in‚0ÁÁ"2“« ‚(:ˆ ¦óöÒÿÄ”Þ gpu.0.gpu‚95.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚275.586' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 » ù¬ ‚(:‚ µóöÒ¸ß‡Þ cpu‚25.687951065206523 proc.cpu.threads‚17) disk./.usagePercent‚76.42378491039987 disk.md0.out‚ 62.6953125% disk.nvme0n1.out‚12479.7158203125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.99392318725586# proc.memory.rssMB‚ 2902.58203125* proc.memory.percent‚17.715954780578613 network.sent‚ 3.726656e+06 network.recv‚607032 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚17785.9169921875 disk.nvme0c0n1.in‚7.1875' disk.nvme0c0n1.out‚12479.7158203125 disk.rbd0.in‚0% disk.nvme1n1.out‚8491.39404296875$ disk./.usageGB‚1173.296157836914 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚3051.06396484375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚8491.39404296875 disk.rbd1.in‚0 disk.nvme0n1.in‚7.1875& proc.memory.availableMB‚ 492.515625ø¹¯×ž­ ‚(:“ µóöÒÊžçÞ gpu.0.gpu‚42.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.355' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.11833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Q‰?2® - _wandbruntime‚2254  _runtime‚2254º“ßȯ  ' lr-AdamW/pg1‚0.00017423099495453425 ' lr-AdamW/pg2‚0.00017423099495453425 trainer/global_step‚2399 $ _timestamp‚1.784527290475281e+09 _runtime‚2254.743419085 _step‚193yû¤úÍ° Ç trainer/global_step‚2399 $ _timestamp‚1.784527290475281e+09 _runtime‚2254.743419085 _step‚193 ' lr-AdamW/pg1‚0.00017423099495453425 ' lr-AdamW/pg2‚0.00017423099495453425Á(!»2± - _wandbruntime‚2254  _runtime‚2254ØôÇóÞ'² ØO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79873771476480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368376103750656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23667990 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368375745287168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19321145 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23678133 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82659652409856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4187793 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2785880933376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366240880967610 $ _timestamp‚1.784527290476288e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365606116610560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23667990 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2769645715968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2765793459748 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9904037 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5715939 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4187796 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365606116610560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2769987140096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366241238568244 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2765452898474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19321143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79873790088704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2785992494080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368376103750656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2769987140096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19321145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _step‚194 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19321143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23678133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2769645715968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82659782582784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368375745287168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799ž ³ ÝO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42999278 _runtime‚2254.744611467 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚363475428069136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚363475445108496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5716241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42989133 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365606099571200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9903732 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366240880967610 $ _timestamp‚1.7845272904s0hÕ!76288e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365606116610560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23667990 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2769645715968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2765793459748 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9904037 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5715939 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4187796 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365606116610560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2769987140096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366241238568244 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2765452898474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19321143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79873790088704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2785992494080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368376103750656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2769987140096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19321145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _step‚194 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19321143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23678133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2769645715968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82659782582784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368375745287168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79873771476480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368376103750656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23667990 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368375745287168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19321145 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23678133 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82659652409856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4187793 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2785880933376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9907772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365754374073344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43016741 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2786996241408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366389420023334 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 $ _timestamp‚1.784527291112108e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5718245 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19328938 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368525488358912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368525129888256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43006582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2770755814912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79906951746560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9908060 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚363622875775968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9907772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365754374073344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43016741 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2786996241408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366389420023334 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 $ _timestamp‚1.784527291112108e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5718245 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19328938 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368525488358912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368525129888256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43006582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2770755814912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79906951746560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9908060 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚363622875775968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157ÃÄé›2· - _wandbruntime‚2255  _runtime‚2255[ θ ÿ /loss/FBt1/detection_loss‚0.5901314166509949 /loss/FBt2/detection_loss‚1.2116424394771457 epoch‚5 _runtime‚2255.382059683 %loss/total_loss‚9.917559332600977 .loss/FRt2/assignment_loss‚2.17412169342182 2loss/SRqqt2/assignment_loss‚0.6287480413902999 1loss/FBt1/assignment_loss‚0.07103244940244592 /loss/FRt2/detection_loss‚0.5617011762028679 trainer/global_step‚2399 % _timestamp‚1.7845272911136122e+09 _step‚196 1loss/SRqqt1/detection_loss‚0.6241305670910786 /loss/FRt1/assignment_loss‚0.920898548437162 2loss/SRqqt1/assignment_loss‚0.6054635212742014 2loss/FBt2/assignment_loss‚0.011048655057866967 1loss/SRqqt2/detection_loss‚1.1285407222527692 /loss/FRt1/detection_loss‚1.3901001019423258ý" ¹ „ /loss/FRt2/detection_loss‚0.5617011762028679 trainer/global_step‚2399 % _timestamp‚1.7845272911136122e+09 _step‚196 1loss/SRqqt1/detection_loss‚0.6241305670910786 /loss/FRt1/assignment_loss‚0.920898548437162 2loss/SRqqt1/assignment_loss‚0.6054635212742014 2loss/FBt2/assignment_loss‚0.011048655057866967 1loss/SRqqt2/detection_loss‚1.1285407222527692 /loss/FRt1/detection_loss‚1.3901001019423258 /loss/FBt1/detection_loss‚0.5901314166509949 /loss/FBt2/detection_loss‚1.2116424394771457 epoch‚5 _runtime‚2255.382059683 %loss/total_loss‚9.917559332600977 .loss/FRt2/assignment_loss‚2.17412169342182 2loss/SRqqt2/assignment_loss‚0.6287480413902999 1loss/FBt1/assignment_loss‚0.07103244940244592ÄØüJ”º ‚(:‰ ÄóöÒÞ«¹Þ disk.dm-1.in‚0( disk.nvme0c0n1.out‚12481.80029296875( proc.memory.availableMB‚ 491.14453125 cpu‚25.697335741193463# disk.dm-0.out‚17787.69189453125 network.sent‚ 3.745494e+06% disk./.usageGB‚1173.2961120605469 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚8492.34423828125 disk.md0.out‚ 63.22265625" disk.dm-1.out‚3051.18896484375 disk.nvme0c0n1.in‚7.1875 disk.rbd1.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0$ memory_percent‚97.00229167938232* proc.memory.percent‚17.724013328552246 proc.cpu.threads‚17 network.recv‚610153 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.in‚0& disk.nvme0n1.out‚12481.80029296875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2903.90234375) disk./.usagePercent‚76.42378192871173' disk.nvme1c1n1.out‚8492.34423828125 disk.rbd1.out‚0 disk.dm-0.in‚ 8.64453125yÓEšž» ‚(:“ ÄóöÒšÚûÞ gpu.0.gpu‚96.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚279.733' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.24433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µŽEÙ2¼ - _wandbruntime‚2272  _runtime‚2272;µ¡-2½ ‘{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9924104809761047 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1752185821533203 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16191570460796356 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17254997789859772 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22007420659065247 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2097937911748886 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6512898802757263 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6165413856506348 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3794104754924774 =&Purity/1FRt*SRqqt*FBt/ev¾¤µe9ent_proportion‚0.6297432780265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 .particle/accuracy_4_of_4‚0.950350821018219 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16937898099422455 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.68455970287323 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868805408477783 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3618144989013672 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8546478748321533 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8757331371307373 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4895907938480377 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.806198000907898 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9708846807479858 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04572281613945961 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6054881811141968 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5352757573127747 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17812779545783997 +particle/sensitivity‚0.6243208646774292 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6467959880828857 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9138060212135315 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3633502423763275 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03652174025774002 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.48375004529953003 (jet/accuracy_1_of_1‚0.32296222448349 *jet/accuracy_2_of_4‚0.7591197490692139 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7292956113815308 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.032170094549655914 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994232177734375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08013223856687546 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2440757006406784 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2338682860136032 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25028538703918457 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.237357497215271 particle/accuracy_2_of_2‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5291416049003601 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5793127417564392 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3149268627166748 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7426682114601135 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24070923030376434 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22508616745471954 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6943556070327759 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9080683588981628 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981736540794373 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03764792159199715 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6652193069458008 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 (particle/accuracy‚0.8471331000328064 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995032548904419 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9143969416618347 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6005898714065552 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6932820677757263 *loss/val_total_loss‚10.228677151836473 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14044176042079926 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2798926830291748 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5614258646965027 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981736540794373 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9854422807693481 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6602856516838074 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23462775349617004 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16274791955947876 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14979158341884613 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25147199630737305 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5881602764129639 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6114481091499329 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6183282732963562 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24391937255859375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.06405407190322876 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_4_of_5‚1 trainer/global_step‚2399 _step‚197 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4366609454154968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6072917580604553 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833432674408 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6904916167259216 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4852963387966156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3633856475353241 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7748925089836121 particle/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967597723007202 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9805741906166077 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6931737661361694 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6865457892417908 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21983453631401062 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20992200076580048 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7310322523117065 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.694692075252533 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1257731318473816 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1276107132434845 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9968253374099731 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2443804293870926 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23462775349617004 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 _runtime‚2272.404097316 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886939525604248 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *jet/accuracy_1_of_3‚0.9088576436042786 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9935805797576904 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5347544550895691 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6124283075332642 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8542664051055908 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.647405743598938 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4086275100708008 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9864100217819214 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5537146925926208 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07312732934951782 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.792101263999939 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.529136061668396 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22141435742378235 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5321569442749023 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6077539920806885 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6890565156936646 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20800110697746277 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43287038803100586 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8140967488288879 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7177945971488953 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10232818871736526 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10232818871736526 % _timestamp‚1.7845273081352918e+09 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29393064975738525 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951339960098267 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9931231141090393 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6883934736251831 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5234375 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977659583091736 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9813360571861267 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11863859742879868 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11968959867954254 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4131253957748413 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6122673749923706 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6114481091499329 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3904660940170288 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7743008732795715 +jet/accuracy_4_of_4‚0.01628045178949833 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 epoch‚5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6386607885360718 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924448132514954 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18653549253940582 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6368997693061829 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.062439899891614914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7800076007843018 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 )jet/accuracy_2_of_3‚0.526511013507843 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3819444477558136 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43239161372184753 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7748925089836121 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.643159806728363 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22722680866718292 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 *jet/accuracy_1_of_4‚0.9714482426643372 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9995090961456299tNœ=¾ –{ @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14044176042079926 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2798926830291748 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5614258646965027 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981736540794373 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9854422807693481 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6602856516838074 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23462775349617004 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16274791955947876 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14979158341884613 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25147199630737305 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5881602764129639 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6114481091499329 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6183282732963562 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24391937255859375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.06405407190322876 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_4_of_5‚1 trainer/global_step‚2399 _step‚197 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4366609454154968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6072917580604553 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833432674408 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6904916167259216 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4852963387966156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3633856475353241 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7748925089836121 particle/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967597723007202 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9805741906166077 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6931737661361694 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6865457892417908 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21983453631401062 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20992200076580048 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7310322523117065 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.694692075252533 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1257731318473816 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1276107132434845 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9968253374099731 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2443804293870926 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23462775349617004 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 _runtime‚2272.404097316 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886939525604248 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *jet/accuracy_1_of_3‚0.9088576436042786 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9935805797576904 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5347544550895691 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6124283075332642 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8542664051055908 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.647405743598938 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4086275100708008 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9864100217819214 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5537146925926208 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07312732934951782 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.792101263999939 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.529136061668396 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22141435742378235 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5321569442749023 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6077539920806885 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6890565156936646 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20800110697746277 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43287038803100586 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8140967488288879 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7177945971488953 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10232818871736526 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10232818871736526 % _timestamp‚1.7845273081352918e+09 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29393064975738525 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951339960098267 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9931231141090393 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6883934736251831 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5234375 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977659583091736 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9813360571861267 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11863859742879868 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11968959867954254 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4131253957748413 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6122673749923706 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6114481091499329 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3904660940170288 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7743008732795715 +jet/accuracy_4_of_4‚0.01628045178949833 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 epoch‚5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6386607885360718 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924448132514954 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18653549253940582 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6368997693061829 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.062439899891614914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7800076007843018 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 )jet/accuracy_2_of_3‚0.526511013507843 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3819444477558136 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43239161372184753 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7748925089836121 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.643159806728363 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22722680866718292 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 *jet/accuracy_1_of_4‚0.9714482426643372 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9995090961456299 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9924104809761047 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1752185821533203 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16191570460796356 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17254997789859772 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22007420659065247 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2097937911748886 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6512898802757263 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6165413856506348 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3794104754924774 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 .particle/accuracy_4_of_4‚0.950350821018219 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16937898099422455 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.68455970287323 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868805408477783 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3618144989013672 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8546478748321533 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8757331371307373 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4895907938480377 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.806198000907898 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9708846807479858 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04572281613945961 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6054881811141968 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5352757573127747 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17812779545783997 +particle/sensitivity‚0.6243208646774292 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6467959880828857 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9138060212135315 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3633502423763275 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03652174025774002 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.48375004529953003 (jet/accuracy_1_of_1‚0.32296222448349 *jet/accuracy_2_of_4‚0.7591197490692139 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7292956113815308 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.032170094549655914 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994232177734375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08013223856687546 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2440757006406784 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2338682860136032 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25028538703918457 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.237357497215271 particle/accuracy_2_of_2‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5291416049003601 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5793127417564392 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3149268627166748 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7426682114601135 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24070923030376434 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22508616745471954 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6943556070327759 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9080683588981628 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981736540794373 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03764792159199715 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6652193069458008 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 (particle/accuracy‚0.8471331000328064 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995032548904419 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9143969416618347 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6005898714065552 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6932820677757263 *loss/val_total_loss‚10.228677151836473Å¥9¸S¿ ‚(:‚ ÓóöÒýœ»Þ$ memory_percent‚95.48637866973877* proc.memory.percent‚17.727351188659668 cpu‚25.702687332150735& disk.nvme0n1.out‚12483.30810546875 network.recv‚614412( disk./.usagePercent‚76.4238403201045 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125 disk.dm-1.out‚3296.607421875( proc.memory.availableMB‚ 739.51171875 proc.cpu.threads‚17 disk.md0.out‚ 63.4765625# disk.dm-0.out‚17789.04345703125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme0n1.in‚7.1875% disk./.usageGB‚1173.2970085144043 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚12483.30810546875 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125# proc.memory.rssMB‚ 2904.44921875 network.sent‚ 3.80668e+06 disk.rbd0.in‚0& disk.nvme1c1n1.out‚8738.1611328125$ disk.nvme1n1.out‚8738.1611328125쬌ŽÀ ‚(:’ ÓóöÒÆáøÞ gpu.0.gpu‚60.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚282.38' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.12666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘nŒÁ ‚(: âóöÒ·¸âÝ disk.md0.in‚0 disk.dm-0.in‚ 8.64453125 disk.dm-1.out‚3296.681640625( disk.nvme0c0n1.out‚12485.53466796875% disk.nvme1c1n1.out‚8738.962890625% disk./.usageGB‚1173.2970275878906 disk.md0.out‚ 63.859375# disk.dm-0.out‚17791.15966796875 disk.dm-1.in‚0& disk.nvme0n1.out‚12485.53466796875 disk.rbd1.in‚0# disk.nvme1n1.out‚8738.962890625* proc.memory.percent‚17.727351188659668 network.sent‚ 3.850728e+06) disk./.usagePercent‚76.42384156247456 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.48799991607666( proc.memory.availableMB‚ 739.24609375# proc.memory.rssMB‚ 2904.44921875 network.recv‚618276 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 cpu‚25.702686055938788 proc.cpu.threads‚17ªpY« ‚(:‘ âóöÒ–ÎøÝ gp½_2"}u.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚195.266' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.08866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aŠà ‚(:ÿ ñóöÒŽ˜…Þ& disk.nvme1c1n1.out‚8985.0068359375$ disk.nvme1n1.out‚8985.0068359375 network.sent‚ 3.855594e+06) disk./.usagePercent‚76.42385125296101# disk.dm-0.out‚17791.83154296875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0$ disk./.usageGB‚1173.297176361084" disk.dm-1.out‚3542.26904296875 disk.nvme0c0n1.in‚7.1875' disk.nvme0c0n1.out‚12486.4833984375' proc.memory.availableMB‚ 739.4296875# proc.memory.rssMB‚ 2904.44921875 cpu‚25.702877388055732 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.48687934875488* proc.memory.percent‚17.727351188659668 network.recv‚621515 disk.md0.out‚64.1875 disk.nvme0n1.in‚7.1875% disk.nvme0n1.out‚12486.4833984375 disk.rbd0.out‚0z\ŒžÄ ‚(:“ ñóöÒÁøÝÞ gpu.0.gpu‚90.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.025' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.34166666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ0j‹2Å - _wandbruntime‚2318  _runtime‚2318 >w°ÉÆ Ã ' lr-AdamW/pg1‚0.00017400263921111696 ' lr-AdamW/pg2‚0.00017400263921111696 trainer/global_step‚2449 % _timestamp‚1.7845273541891527e+09 _runtime‚2318.457301308 _step‚198$Aü‹ÎÇ È ' lr-AdamW/pg1‚0.00017400263921111696 ' lr-AdamW/pg2‚0.00017400263921111696 trainer/global_step‚2449 % _timestamp‚1.7845273541891527e+09 _runtime‚2318.457301308 _step‚198Æ\Ä&õ2È - _wandbruntime‚2318  _runtime‚2318Šè‡¾á'É ÛO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374962168553984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10153891 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84417822367744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377808059673600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19833601 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24244284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2841580522058 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19833601 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24234141 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377807701210112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44077887 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4278612 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377807701210112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81555382255104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚375609457411744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374962168553984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372768217450960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚2449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2862458724864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2862559799808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚375609815012378 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19833603 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10154178 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24234141 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374962185593344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374962185593344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2845532656128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875566 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377807701210112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81555382255104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚375609457411744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374962168553984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372768217450960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚2449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2862458724864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2862559799808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚375609815012378 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19833603 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10154178 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24234141 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374962185593344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374962185593344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2845532656128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875566 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2846984186368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2846642755072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19841396 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2846642755072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2842348349036 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24253954 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84452123184640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚375757996467468 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44085191 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19841394 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚2319.091084051 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19841396 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375110460095488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377957444281856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10158189 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375110460095488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81588543912960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10157908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377957085811200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44095350 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24243797 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81588562525184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24243797 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5877843 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845273548227603e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚372915665157792 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84452236573184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚2449JävÙ)Í éM ZMDeviceStatsMonitor.on_train_bat†XèÆ&ch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845273548227603e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚372915665157792 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84452236573184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚2449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2846984186368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2846642755072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19841396 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2846642755072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2842348349036 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24253954 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84452123184640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚375757996467468 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44085191 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19841394 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚2319.091084051 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19841396 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375110460095488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377957444281856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10158189 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375110460095488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81588543912960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10157908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377957085811200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44095350 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24243797 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81588562525184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24243797 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5877843 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157Èd!tr2Î - _wandbruntime‚2319  _runtime‚2319”œ‘ Ï ƒ 0loss/FRt1/assignment_loss‚0.8889420135800107 0loss/FRt2/assignment_loss‚2.1951590907850345 1loss/SRqqt1/detection_loss‚0.6762747168128194 /loss/FBt1/detection_loss‚0.5776412586921907 &loss/total_loss‚10.192369735626162 _step‚201 2loss/SRqqt1/assignment_loss‚0.5690768362225441 /loss/FRt2/detection_loss‚0.5769510253433621 epoch‚6 trainer/global_step‚2449 % _timestamp‚1.7845273548238845e+09 /loss/FRt1/detection_loss‚1.5159593681541395 0loss/SRqqt2/detection_loss‚1.057473775574566 .loss/FBt2/detection_loss‚1.423042144825948 _runtime‚2319.095548008 2loss/SRqqt2/assignment_loss‚0.6534118411023583 1loss/FBt1/assignment_loss‚0.05814228103871182 4loss/FBt2/assignment_loss‚0.00029538349447680697sŽÐ ˆ 2loss/SRqqt2/assignment_loss‚0.6534118411023583 1loss/FBt1/assignment_loss‚0.05814228103871182 4loss/FBt2/assignment_loss‚0.00029538349447680697 0loss/FRt1/assignment_loss‚0.8889420135800107 0loss/FRt2/assignment_loss‚2.1951590907850345 1loss/SRqqt1/detection_loss‚0.6762747168128194 /loss/FBt1/detection_loss‚0.5776412586921907 &loss/total_loss‚10.192369735626162 _step‚201 2loss/SRqqt1/assignment_loss‚0.5690768362225441 /loss/FRt2/detection_loss‚0.5769510253433621 epoch‚6 trainer/global_step‚2449 % _timestamp‚1.7845273548238845e+09 /loss/FRt1/detection_loss‚1.5159593681541395 0loss/SRqqt2/detection_loss‚1.057473775574566 .loss/FBt2/detection_loss‚1.423042144825948 _runtime‚2319.095548008Ɇ€f‰Ñ ‚(:„ €ôöÒÔºÔÞ$ disk./.usageGB‚1173.297176361084 disk.md0.out‚ 64.578125" disk.dm-1.out‚3542.31982421875 disk.nvme0n1.in‚7.1875' disk.nvme1c1n1.out‚8985.85595703125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2904.44921875 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚17794.39599609375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0# memory_percent‚95.4888105392456* proc.memory.percent‚17.727351188659668 proc.cpu.threads‚17) disk./.usagePercent‚76.42385125296101% disk.nvme1n1.out‚8985.85595703125( disk.nvme0c0n1.out‚12489.08837890625 disk.rbd0.out‚0 network.recv‚624651 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0& disk.nvme0n1.out‚12489.08837890625 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0( proc.memory.availableMB‚ 739.11328125 cpu‚25.70199044805478 network.sent‚ 3.874473e+06:Ú^ØžÒ ‚(:“ €ôöÒ¼‹˜ß gpu.0.gpu‚83.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.015' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.33833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¯Á“Ó ‚(:ˆ ôöÒƸ©Þ cpu‚25.700075788639023 proc.cpu.threads‚17 network.recv‚629208 disk.md0.in‚0# disk.dm-0.out‚17795.23583984375 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚8986.41552734375 disk.rbd1.in‚0% disk./.usageGB‚1173.2966270446777 disk.md0.out‚ 64.7109375" disk.dm-1.out‚3542.38623046875( disk.nvme0c0n1.out‚12489.74951171875& disk.nvme0n1.out‚12489.74951171875 disk.rbd0.out‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125) disk./.usagePercent‚76.42381547270331 disk.dm-1.in‚0$ memory_percent‚95.48923969268799( proc.memory.availableMB‚ 739.04296875# proc.memory.rssMB‚ 2904.44921875* proc.memory.percent‚17.727351188659668 network.sent‚ 3.923139e+06 disk.dm-0.in‚ 8.64453125 disk.rbd0.in‚0% disk.nvme1n1.out‚8986.41552734375§¹Ç%žÔ ‚(:“ ôöÒ˜âüÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚127.027' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.34233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e;¾ò…Õ ‚(:ú žôöÒúÛ˜Þ disk.rbd1.out‚0 disk.dm-0.in‚ 8.64453125( proc.memory.availableMB‚ 738.85546875) disk./.usagePercent‚76.42384429568868& disk.nvme0c0n1.out‚12492.119140625 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚12492.119140625 disk.md0.in‚0! disk.dm-0.out‚17797.666015625# proc.memory.rssMB‚ 2904.44921875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 3542.49609375 disk.md0.out‚ 64.91796875$ memory_percent‚95.49038410186768* proc.memory.percent‚17.727351188659668 network.recv‚632724% disk./.usageGB‚1173.2970695495605 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.45703125" disk.nvme1n1.out‚ 8987.04296875 cpu‚25.698391463742915 network.sent‚ 3.929192e+06 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125$ disk.nvme1c1n1.out‚ 8987.04296875 disk.rbd1.in‚0 òwqžÖ ‚(:“ žôöÒ…ÃèÞ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚269.202' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.73400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«æU2× - _wandbruntime‚2363  _runtime‚2363ñöë0ÇØ Á _runtime‚2363.910539863 _step‚202 & lr-AdamW/pg1‚0.0001737688489687796 & lr-AdamW/pg2‚0.0001737688489687796 trainer/global_step‚2499 % _timestamp‚1.7845273996420243e+09J‚¨ñÌÙ Æ & lr-AdamW/pg2‚0.0001737688489687796 trainer/global_step‚2499 % _timestamp‚1.7845273996420243e+09 _runtime‚2363.910539863 _step‚202 & lr-AdamW/pg1‚0.0001737688489687796Ê)^+2Ú - _wandbruntime‚2363  _runtime‚2363«TËá'Û ÛO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚382375910700544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 trainer/global_step‚2499 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44942044 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86132746083840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383036831599386 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2896723174192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20223251 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83214395759104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382375893661184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382375893661184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2901101757952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24718793 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10355985 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚203 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚385277353882624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2901443182080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2901101757952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382375910700544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4365309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5990673 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83214377146880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚385277353882624 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383036473998752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24728936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2918476303360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2363.910740933 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24728936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 % _timestamp‚1.7845273996426978e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚385276995419136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20223253 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4365312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86132872062464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20223253 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2901443182080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5990389 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385276995419136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44952189 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20223251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2897063735466 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383036473998752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24728936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2918476303360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2363.910740933 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24728936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 % _timestamp‚1.7845273996426978e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚385276995419136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20223253 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4365312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86132872062464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20223253 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2901443182080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5990389 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385276995419136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44952189 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20223251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2897063735466 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 trainer/global_step‚2499 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44942044 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86132746083840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383036831599386 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2896723174192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20223251 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83214395759104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382375893661184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382375893661184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2901101757952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24718793 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10355985 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚203 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚385277353882624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2901443182080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2901101757952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382375910700544ËâàN2Ý - _wandbruntime‚2364  _runtime‚23647”#né&Þ ãM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2898172123830 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 2364.5463017 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20231044 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24738606 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5992671 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83247557416960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5992962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 _step‚204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380287198531392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382524185202688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383185370655222 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10359714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2902553288192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2919481140224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2499 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2902211856896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44969652 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24728449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382524168163328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383185013054476 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845274002780006e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382524168163328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385426738490880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20231046 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10360008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20231044 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚385426738490880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2902211856896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44969652 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4367043 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83247576029184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44959493 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24728449 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452“¦î&ß èM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382524168163328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385426738490880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20231046 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10360008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20231044 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚385426738490880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2902211856896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44969652 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4367043 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83247576029184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44959493 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24728449 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2898172123830 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 2364.5463017 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20231044 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24738606 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5992671 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83247557416960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5992962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 _step‚204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380287198531392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382524185202688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383185370655222 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10359714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2902553288192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2919481140224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2499 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2902211856896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44969652 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24728449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382524168163328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383185013054476 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845274002780006e+09̦véB2à - _wandbruntime‚2364  _runtime‚2364íWá ý 2loss/SRqqt1/assignment_loss‚0.6023034275131276 0loss/SRqqt2/assignment_loss‚0.62518381606759 1loss/FBt1/assignment_loss‚0.07759285276840074 1loss/SRqqt2/detection_loss‚1.0376326055076361 .loss/FBt1/detection_loss‚0.587866476053197 _step‚205 /loss/FRt1/assignment_loss‚0.892552048999763 /loss/FRt1/detection_loss‚1.4546298125973105 /loss/FRt2/detection_loss‚0.5283227364914387 1loss/SRqqt1/detection_loss‚0.6571921881523786 .loss/FBt2/detection_loss‚1.183902784367092 trainer/global_step‚2499 % _timestamp‚1.7845274002791605e+09 _runtime‚2364.546977191 0loss/FRt2/assignment_loss‚2.2756396601439786 %loss/total_loss‚9.927982298983547 2loss/FBt2/assignment_loss‚0.005163890321634224 epoch‚6Ašþçâ ‚ 2loss/SRqqt1/assignment_loss‚0.6023034275131276 0loss/SRqqt2/assignment_loss‚0.62518381606759 1loss/FBt1/assignment_loss‚0.07759285276840074 1loss/SRqqt2/detection_loss‚1.0376326055076361 .loss/FBt1/detection_loss‚0.587866476053197 _step‚205 /loss/FRt1/assignment_loss‚0.892552048999763 /loss/FRt1/detection_loss‚1.4546298125973105 /loss/FRt2/detection_loss‚0.5283227364914387 1loss/SRqqt1/detection_loss‚0.6571921881523786 .loss/FBt2/detection_loss‚1.183902784367092 trainer/global_step‚2499 % _timestamp‚1.7845274002791605e+09 _runtime‚2364.546977191 0loss/FRt2/assignment_loss‚2.2756396601439786 %loss/total_loss‚9.927982298983547 2loss/FBt2/assignment_loss‚0.005163890321634224 epoch‚6Í.î%zã ‚(:ï ­ôöÒ°ýùÝ proc.memory.availableMB‚739 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme1c1n1.out‚8987.619140625 cpu‚25.696521336891955 network.sent‚ 3.932003e+06 disk.nvme1n1.in‚ 1.45703125% disk.nvme0c0n1.out‚12493.10546875 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0# proc.memory.rssMB‚ 2904.44921875* proc.memory.percent‚17.727351188659668 proc.cpu.threads‚17) disk./.usagePercent‚76.42384578653275% disk./.usageGB‚1173.2970924377441# disk.nvme1n1.out‚8987.619140625 disk.md0.in‚0! disk.dm-0.out‚17798.373046875 disk.dm-1.out‚ 3542.57421875 disk.nvme0c0n1.in‚7.1875 disk.rbd1.in‚0 network.recv‚635211 disk.md0.out‚ 65.27734375 disk.dm-0.in‚ 8.64453125 disk.nvme0n1.in‚7.1875# disk.nvme0n1.out‚12493.10546875 disk.rbd0.in‚0" memory_percent‚95.489501953125‚„.ä ‚(:’ ­ôöÒ’¨òÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.823' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.940999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a ix‰å ‚(:þ ¼ôöÒÒþ÷Ý network.recv‚639191% disk./.usageGB‚1173.2971496582031 disk.rbd1.out‚0 disk.md0.out‚ 65.69140625 disk.nvme0n1.in‚7.1875$ memory_percent‚95.48964500427246 disk.nvme1c1n1.in‚ 1.45703125 disk.dm-1.out‚3542.658203125$ disk.nvme0c0n1.out‚ 12496.8828125" disk.nvme0n1.out‚ 12496.8828125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2904.44921875 cpu‚25.695085849877835) disk./.usagePercent‚76.42384951364292' disk.nvme1c1n1.out‚8988.65087890625 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚17802.20263671875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0' proc.memory.availableMB‚ 738.9765625 network.sent‚ 3.994327e+06 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚8988.65087890625 disk.md0.in‚0* proc.memory.percent‚17.727351188659668 proc.cpu.threads‚17pz¡žæ ‚(:“ ¼ôöÒòÞËÞ gpu.0.gpu‚54.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚181.407' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.46900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰$‡ç ‚(:ü ËôöÒ‚ýÞ% disk./.usageGB‚1173.2966651916504 disk.md0.in‚0 disk.md0.out‚ 65.984375 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875% disk.nvme1c1n1.out‚8989.119140625 disk.nvme1n1.in‚ 1.45703125# disk.nvme1n1.out‚8989.119140625 network.sent‚ 3.99921e+06" disk.dm-0.out‚17803.1162109375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚12498.0029296875% disk.nvme0n1.out‚12498.0029296875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 cpu‚25.693456613009428) disk./.usagePercent‚76.42381795744345 disk.dm-0.in‚ 8.64453125 disk.dm-1.out‚3542.701171875 disk.rbd1.in‚0$ memory_percent‚95.48916816711426# proc.memory.rssMB‚ 2904.44921875* proc.memory.percent‚17.727351188659668 network.recv‚642956 disk.rbd0.in‚0 disk.rbd1.out‚0' proc.memory.availableMB‚ 739.0546875 proc.cpu.threads‚17~šXf“è ‚(:ˆ ËôöÒâ’ÞÞ gpu.0.gpu‚48.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚134.115' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚44.705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îÏ'v2é - _wandbruntime‚2408  _runtime‚2408 g4Çê Á _runtime‚2408.879558576 _step‚206 & lr-AdamW/pg1‚0.0001735296392434326 & lr-AdamW/pg2‚0.0001735296392434326 trainer/global_step‚2549 % _timestamp‚1.7845274446106982e+09ƒh4jÌë Æ & lr-AdamW/pg2‚0.0001735296392434326 trainer/global_step‚2549 % _timestamp‚1.7845274446106982e+09 _runtime‚2408.879558576 _step‚206 & lr-AdamW/pg1‚0.0001735296392434326ÎrVŒ©2ì - _wandbruntime‚2408  _runtime‚2408ÍAˆ/ í ÛO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2956670859776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45826491 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25203445 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25213588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6106317 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387511284198160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2957012283904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84873390650880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2952206387600 _runtime‚2408.879643986 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974238794240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87847629445120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974352452096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2957012283904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10558323†|.²9 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84873409263104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2956670859776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20612903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚2549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20612901 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389789635807744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392746289628160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25203445 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392746648091648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4452012 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20612903 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25213588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390463848186394 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4452009 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389789618768384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6106025 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392746648091648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2952546948874 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392746289628160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87847761715200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387511301237520pó$æ'î àO _runtime‚2408.879643986 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974238794240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87847629445120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974352452096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2957012283904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10558329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84873409263104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2956670859776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20612903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚2549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20612901 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389789635807744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392746289628160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25203445 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392746648091648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4452012 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20612903 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25213588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390463848186394 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4452009 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389789618768384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6106025 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392746648091648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2952546948874 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392746289628160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87847761715200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387511301237520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2956670859776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45826491 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25203445 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25213588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6106317 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387511284198160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2957012283904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84873390650880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2952206387600ÏÖ*uñ2ï - _wandbruntime‚2409  _runtime‚2409F³Íë&ð åM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387658731904992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚392896032699904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20620694 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4453743 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4453746 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10562012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975466376704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389937893270528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20620696 _step‚208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392895674229248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25223258 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45843954 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10562301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚2409.514768428 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚392896032699904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20620694 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4453743 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.currenåmlÏÈt‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4453746 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10562012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975466376704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389937893270528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20620696 _step‚208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392895674229248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25223258 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45843954 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10562301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚2409.514768428 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387658731904992 EKðƒù ‚(:ø øôöÒöߨÞ disk.dm-1.in‚0 disk.dm-1.out‚ 3542.9921875& disk.nvme0c0n1.out‚12503.435546875 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚95.49000263214111* proc.memory.percent‚17.727351188659668 cpu‚25.68879841394129 network.sent‚ 4.073386e+06 disk.md0.out‚ 66.73046875 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚ 17808.2734375 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0# disk.nvme1n1.out‚8990.787109375# proc.memory.rssMB‚ 2904.44921875 proc.cpu.threads‚17 network.recv‚653919) disk./.usagePercent‚76.42387684578422% disk./.usageGB‚1173.2975692749023$ disk.nvme0n1.out‚12503.435546875% disk.nvme1c1n1.out‚8990.787109375 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125( proc.memory.availableMB‚ 738.91796875 disk.md0.in‚0L§\‡ú ‚(:’ øôöÒÊ¡çÞ gpu.0.gpu‚14.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TI2û - _wandbruntime‚2454  _runtime‚2454pc{Éü à ' lr-AdamW/pg1‚0.00017328502539906964 ' lr-AdamW/pg2‚0.00017328502539906964 trainer/global_step‚2599 % _timestamp‚1.7845274899702253e+09 _runtime‚2454.238414076 _step‚210*O 'Îý È ' lr-AdamW/pg1‚0.00017328502539906964 ' lr-AdamW/pg2‚0.00017328502539906964 trainer/global_step‚2599 % _timestamp‚1.7845274899702253e+09 _runtime‚2454.238414076 _step‚210ÒÈÐ&Þ2þ - _wandbruntime‚2454  _runtime‚2454#ü¶à'ÿ ÚO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397203343875584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚400215583837184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25688097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394882817571760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46700793 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3030179871232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21002551 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3012239961600 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397203360914944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6221545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397203343875584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10759969 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25698240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25698240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚397890507172768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3007689601008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚394882834611120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400215942300672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10760257 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4538709 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89562602638336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89562489242624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 % _timestamp‚1.7845274899713452e+09 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4538712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400215583837184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3012581385728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3012239961600 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21002553 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25688097 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6221260 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452€Xgå'€ ßO _step‚211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397203343875584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10759969 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25698240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25698240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚397890507172768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3007689601008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚394882834611120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400215942300672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10760257 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4538709 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89562602638336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89562489242624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 % _timestamp‚1.7845274899713452e+09 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4538712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400215583837184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3012581385728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3012239961600 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21002553 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25688097 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6221260 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397203343875584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚400215583837184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25688097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394882817571760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46700793 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3030179871232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21002551 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3012239961600 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397203360914944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6221545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232ÓÈãc¨2 - _wandbruntime‚2454  _runtime‚2454/±%%‚ ãM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25707910 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78452749060339e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397351618377728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398039403829238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21010344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21010346 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4540446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3013350060544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86565603037184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397351618377728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46718256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6223807 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3031303395328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395030265278592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3008797989260 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4540443 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400365326908928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚400364968438272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25697753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25697753 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400365326908928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89596780461056 trainer/global_step‚2599 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10763959 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397351635417088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86565584424960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395030248239232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3013350060544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚2454.871722025 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46718256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46708097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398039046228492 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46708097 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3013691491840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397351635417088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ç„/àî&ƒ èM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86565603037184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397351618377728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46718256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6223807 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3031303395328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395030265278592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3008797989260 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4540443 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400365326908928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚400364968438272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25697753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25697753 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400365326908928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89596780461056 trainer/global_step‚2599 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10763959 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397351635417088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86565584424960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395030248239232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3013350060544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚2454.871722025 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46718256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46708097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398039046228492 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46708097 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3013691491840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397351635417088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25707910 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78452749060339e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397351618377728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398039403829238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21010344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21010346 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4540446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3013350060544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656ÔTzÈw2„ - _wandbruntime‚2454  _runtime‚2454OÉ"³… þ 2loss/SRqqt2/assignment_loss‚0.6167368337413045 1loss/FBt1/assignment_loss‚0.10498548110436595 1loss/SRqqt1/detection_loss‚0.6526337125663288 0loss/SRqqt2/detection_loss‚1.009045803478694 /loss/FBt1/detection_loss‚0.5330237686535124 %loss/total_loss‚9.539265634699856 2loss/FBt2/assignment_loss‚0.015683795798943346 _runtime‚2454.872403816 /loss/FRt1/detection_loss‚1.4150085805600134 /loss/FRt2/detection_loss‚0.5864474022643936 .loss/FBt2/detection_loss‚1.079155880565706 epoch‚6 trainer/global_step‚2599 _step‚213 /loss/FRt2/assignment_loss‚2.026478152327199 % _timestamp‚1.7845274906045322e+09 0loss/FRt1/assignment_loss‚0.8813767522686788 1loss/SRqqt1/assignment_loss‚0.618689471370717øî ' † ƒ 2loss/FBt2/assignment_loss‚0.015683795798943346 _runtime‚2454.872403816 /loss/FRt1/detection_loss‚1.4150085805600134 /loss/FRt2/detection_loss‚0.5864474022643936 .loss/FBt2/detection_loss‚1.079155880565706 epoch‚6 trainer/global_step‚2599 _step‚213 /loss/FRt2/assignment_loss‚2.026478152327199 % _timestamp‚1.7845274906045322e+09 0loss/FRt1/assignment_loss‚0.8813767522686788 1loss/SRqqt1/assignment_loss‚0.618689471370717 2loss/SRqqt2/assignment_loss‚0.6167368337413045 1loss/FBt1/assignment_loss‚0.10498548110436595 1loss/SRqqt1/detection_loss‚0.6526337125663288 0loss/SRqqt2/detection_loss‚1.009045803478694 /loss/FBt1/detection_loss‚0.5330237686535124 %loss/total_loss‚9.539265634699856Õ÷Ôì—‡ ‚(:ö ‡õöÒï²ËÞ disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125# disk.nvme1n1.out‚8991.228515625 disk.dm-1.out‚ 3543.05078125$ disk.nvme0n1.out‚12504.529296875# memory_percent‚96.9933032989502 network.recv‚657645) disk./.usagePercent‚76.42387783968027* proc.memory.percent‚17.727351188659668 cpu‚25.687043188344248 network.sent‚ 4.08919e+06 disk.rbd1.in‚0 disk.md0.in‚0 disk.md0.out‚ 66.94921875 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1173.2975845336914% disk.nvme1c1n1.out‚8991.228515625 disk.dm-0.out‚ 17809.265625 disk.nvme0c0n1.in‚7.1875& disk.nvme0c0n1.out‚12504.529296875' proc.memory.availableMB‚ 492.6171875# proc.memory.rssMB‚ 2904.44921875 proc.cpu.threads‚17 disk.dm-0.in‚ 8.64453125Z`X†‘ˆ ‚(:† ‡õöÒÊÕ‘ß gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚274.146' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾.bÿ‰ ‚(:‚ –õöÒŸ¡îÝ disk.md0.out‚ 67.48046875 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0% disk.nvme0n1.out‚12509.2998046875 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚661744 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0' disk.nvme0c0n1.out‚12509.2998046875 disk.rbd1.out‚0$ memory_percent‚96.99492454528809' proc.memory.availableMB‚ 492.3515625* proc.memory.percent‚17.727351188659668 network.sent‚ 4.138605e+06$ disk./.usageGB‚1170.580234527588! disk.dm-0.out‚17813.666015625 disk.nvme0c0n1.in‚7.1875 cpu‚25.685431045640012" disk.dm-1.out‚3543.13427734375 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚8992.07275390625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2904.44921875 proc.cpu.threads‚17( disk./.usagePercent‚76.2468806075615% disk.nvme1n1.out‚8992.07275390625 îÙ“Š ‚(:ˆ –õöÒ±úÃÞ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚278.142' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?W¡SŽ‹ ‚(:ƒ ¥õöÒ‰ÍôÝ) disk./.usagePercent‚76.24687712892533 disk.dm-1.in‚0" disk.dm-1.out‚3543.16162109375 disk.rbd1.out‚0( proc.memory.availableMB‚ 492.34765625# proc.memory.rssMB‚ 2904.44921875 proc.cpu.threads‚17% disk./.usageGB‚1170.5801811218262 disk.nvme1n1.in‚ 1.45703125 disk.md0.out‚ 67.76171875! disk.dm-0.out‚17814.287109375 disk.nvme0c0n1.in‚7.1875' disk.nvme0c0n1.out‚12510.1865234375 disk.nvme0n1.in‚7.1875' disk.nvme1c1n1.out‚8992.45947265625 network.sent‚ 4.144629e+06 network.recv‚665338% disk.nvme1n1.out‚8992.45947265625 disk.md0.in‚0% disk.nvme0n1.out‚12510.1865234375 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0" memory_percent‚96.994948387146 disk.dm-0.in‚ 8.64453125 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.727351188659668 cpu‚25.684120762523104[1à<Œ ‚(:’ ¥õöÒöâ‰Þ gpu.0.gpu‚27.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚255.031' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.01033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ) 2 - _wandbruntime‚2499  _runtime‚2499äå|dÇŽ Á & lr-AdamW/pg2‚0.0001730350231467808 trainer/global_step‚2649 % _timestamp‚1.7845275349738402e+09 _runtime‚2499.241975883 _step‚214 & lr-AdamW/pg1‚0.0001730350231467808¬…CÌ Æ % _timestamp‚1.7845275349738402e+09 _runtime‚2499.241975883 _step‚214 & lr-AdamW/pg1‚0.0001730350231467808 & lr-AdamW/pg2‚0.0001730350231467808 trainer/global_step‚2649Öw(Úì2 - _wandbruntime‚2499  _runtime‚2499Ü E-‘ ÚO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6337219 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3086052655104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407685236509696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404617068982784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚2499.242927094 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88191417658880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405317881360410 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407684878046208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3068150487552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91277488926208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3068150487552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21392201 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6336940 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3085943191552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402254367984720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21392203 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404617086022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404617086022144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47575095 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405317523759776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3063172814416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚2649 L?DeviceStatsMonitor.on_train_batch_start/active.large¼ ì̳_pool.freed‚21392201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26182892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88191436271104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407684878046208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91277360850432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 $ _timestamp‚1.784527534974918e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10962631 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26172749 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4625412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚215 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404617068982784 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4625409(|– å'’ ßO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26182892 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3067809063424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21392201 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6336940 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3085943191552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402254367984720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21392203 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404617086022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404617086022144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47575095 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405317523759776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3063172814416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚2649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21392201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26182892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88191436271104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407684878046208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91277360850432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 $ _timestamp‚1.784527534974918e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10962631 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26172749 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4625412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚215 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404617068982784 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4625409 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6337219 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3086052655104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407685236509696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404617068982784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚2499.242927094 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88191417658880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405317881360410 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407684878046208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3068150487552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91277488926208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3068150487552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91311657517568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47582399 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10966393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404765343484928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402401781612832 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26182405 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3087160656384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21399996 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88224597928960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407834262647296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26182405 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407834621117952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3069260593664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47592558 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405466062815500 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21399996 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26192562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405466420416246 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3068919162368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3064281202668 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91311777197568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3064621764054 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4627143 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6339250 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3087059588608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404765360524288 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88224597928960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407834262647296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26182405 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407834621117952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3069260593664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47592558 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405466062815500 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21399996 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26192562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405466420416246 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3068919162368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3064281202668 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91311777197568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3064621764054 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4627143 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6339250 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3087059588608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404765360524288 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91311657517568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47582399 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10966393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404765343484928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402401781612832 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280Ø™¬ˆ…2– - _wandbruntime‚2500  _runtime‚2500âÀ]þú— û _runtime‚2500.208628309 1loss/SRqqt2/detection_loss‚1.0969544752822198 0loss/FRt1/assignment_loss‚0.8305475500676904 2loss/SRqqt1/assignment_loss‚0.6119722064012687 1loss/FBt1/assignment_loss‚0.04232552624501668 2loss/FBmÚ1t2/assignment_loss‚0.004519230953813442 trainer/global_step‚2649 # _timestamp‚1.78452753594065e+09 /loss/FRt2/assignment_loss‚2.076716281609684 2loss/SRqqt2/assignment_loss‚0.6354670352569354 /loss/FBt1/detection_loss‚0.6236219960131857 epoch‚6 _step‚217 /loss/FRt1/detection_loss‚1.3434030184499932 /loss/FRt2/detection_loss‚0.5531345566966127 0loss/SRqqt1/detection_loss‚0.628485502361457 .loss/FBt2/detection_loss‚1.151856909506023 #loss/total_loss‚9.5990042888439¢Ø™¾˜ € 0loss/FRt1/assignment_loss‚0.8305475500676904 2loss/SRqqt1/assignment_loss‚0.6119722064012687 1loss/FBt1/assignment_loss‚0.04232552624501668 2loss/FBt2/assignment_loss‚0.004519230953813442 trainer/global_step‚2649 # _timestamp‚1.78452753594065e+09 /loss/FRt2/assignment_loss‚2.076716281609684 2loss/SRqqt2/assignment_loss‚0.6354670352569354 /loss/FBt1/detection_loss‚0.6236219960131857 epoch‚6 _step‚217 /loss/FRt1/detection_loss‚1.3434030184499932 /loss/FRt2/detection_loss‚0.5531345566966127 0loss/SRqqt1/detection_loss‚0.628485502361457 .loss/FBt2/detection_loss‚1.151856909506023 #loss/total_loss‚9.5990042888439 _runtime‚2500.208628309 1loss/SRqqt2/detection_loss‚1.0969544752822198ÙLä³Ð†™ ‚(:û ´õöÒÚ÷Ý disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.md0.out‚68.34375 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0$ memory_percent‚96.99532985687256% disk./.usageGB‚1170.5801467895508 disk.rbd0.in‚0& disk.nvme1c1n1.out‚8994.9365234375 disk.md0.in‚0# disk.dm-0.out‚17825.74462890625( proc.memory.availableMB‚ 492.28515625# proc.memory.rssMB‚ 2904.44921875 cpu‚25.682826554345507$ disk.nvme0c0n1.out‚ 12520.6328125 disk.nvme0n1.in‚7.1875" disk.nvme0n1.out‚ 12520.6328125 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0" disk.dm-1.out‚3543.37646484375* proc.memory.percent‚17.727351188659668 network.sent‚ 4.14835e+06 network.recv‚668449 disk.rbd0.out‚0$ disk.nvme1n1.out‚8994.9365234375 proc.cpu.threads‚17) disk./.usagePercent‚76.24687489265922 disk.nvme0c0n1.in‚7.1875h+[î“š ‚(:ˆ ´õöÒ—«¸Þ gpu.0.gpu‚67.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.287' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.429 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!†ÌG‰› ‚(:þ ÃõöÒŒÕƒÞ disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125) disk./.usagePercent‚76.24689129194401& disk.nvme0n1.out‚12521.37255859375' disk.nvme1c1n1.out‚8995.47705078125% disk.nvme1n1.out‚8995.47705078125 disk.md0.in‚0 disk.md0.out‚ 68.5234375# memory_percent‚96.9940185546875# proc.memory.rssMB‚ 2904.44921875# disk.dm-0.out‚17826.60009765625 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚12521.37255859375 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 cpu‚25.68164476745687 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚3543.39208984375 disk.rbd1.out‚0! proc.memory.availableMB‚492.5* proc.memory.percent‚17.727351188659668 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 4.21272e+06 network.recv‚672985% disk./.usageGB‚1170.5803985595703;»Sˆœ ‚(:’ ÃõöÒΓ†Þ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.99' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.66333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ(oü’ ‚(:‡ ÒõöÒ줧Þ disk.md0.in‚0 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚3546.36767578125$ memory_percent‚96.99409008026123( disk.nvme0c0n1.out‚14063.94775390625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.md0.out‚ 68.8671875# disk.dm-0.out‚19362.82275390625# proc.memory.rssMB‚ 2904.44921875 network.sent‚ 4.217683e+06 network.recv‚676256 disk.rbd0.in‚0' disk.nvme1c1n1.out‚8999.46923828125 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0* proc.memory.percent‚17.727351188659668 cpu‚25.68047296121461) disk./.usagePercent‚76.36860082017893% disk./.usageGB‚1172.4489440917969 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚14063.94775390625 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1n1.out‚8999.46923828125( proc.memory.availableMB‚ 492.48828125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚7.1875 disk.rbd1.out‚0(k|0œž ‚(:‘ ÒõöÒƒ´ïÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚276.022' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.00733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k"”2Ÿ - _wandbruntime‚2544  _runtime‚2544ˆàåGÄ  ¾ & lr-AdamW/pg1‚0.0001727796485437436 & lr-AdamW/pg2‚0.0001727796485437436 trainer/global_step‚2699 " _timestamp‚1.7845275802313e+09 _runtime‚2544.499394197 _step‚218s8ßÉ¡ à & lr-AdamW/pg2‚0.0001727796485437436 trainer/global_step‚2699 " _timestamp‚1.7845275802313e+09 _runtime‚2544.499394197 _step‚218 & lr-AdamW/pg1‚0.0001727796485437436Ú/–ߘ2¢ - _wandbruntime‚2544  _runtime‚2544LÅ\—à'£ ÚO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2544.503679794 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415154530718720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11165113 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89850449775104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409625884318960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412030811129344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123719589376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21781853 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141884404736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415154172255232 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26657401 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6453001 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123378165248 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92992216589824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412030794089984 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92992334179840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26667544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123378165248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21781851 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3123719589376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _step‚219 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412030811129344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89850431162880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26667544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48449397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845275802320702e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409625901358320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412030794089984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚412744540346784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3141785426944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11164827 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3118656027824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48439252 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274±ÜQéå'¤ ßO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6452718 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21781851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3118996589098 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26657401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26657401 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6453001 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123378165248 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92992216589824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412030794089984 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92992334179840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26667544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123378165248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21781851 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3123719589376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _step‚219 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412030811129344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89850431162880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26667544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48449397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845275802320702e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409625901358320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412030794089984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚412744540346784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3141785426944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11164827 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3118656027824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48439252 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2544.503679794 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415154530718720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11165113 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89850449775104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409625884318960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412030811129344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123719589376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21781853 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141884404736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415154172255232 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚220 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415303915326976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6455245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409773332025792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3120104977462 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48466860 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21789644 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11169091 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124829695488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412179085631488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4713843 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4713846 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26667057 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93026627301376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93026513912832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3142902479872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3119764416076 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batchnñ‹¼ _end/segment.large_pool.freed‚0 trainer/global_step‚2699 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48456701 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124488264192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6454979 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412179068592128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412893079402508 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚412179068592128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48466860 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415303556856320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚2545.130737657 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11168822 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415303915326976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142997256192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180ö”Fï&§ éM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48456701 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124488264192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6454979 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412179068592128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412893079402508 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚412179068592128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48466860 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415303556856320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚2545.130737657 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11168822 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415303915326976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142997256192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21789646 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89883611432960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚220 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415303915326976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6455245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409773332025792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3120104977462 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48466860 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21789644 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11169091 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124829695488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412179085631488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4713843 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4713846 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26667057 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93026627301376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93026513912832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3142902479872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3119764416076 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2699܆¨™î2¨ - _wandbruntime‚2545  _runtime‚2545_`D © ƒ &loss/total_loss‚10.078727163641675 epoch‚6 2loss/SRqqt2/assignment_loss‚0.7130339602645588 /loss/FRt1/detection_loss‚1.4212562178983903 /loss/FBt2/detection_loss‚1.2958789929266898 2loss/SRqqt1/assignment_loss‚0.5954954509916938 1loss/SRqqt2/detection_loss‚0.9881195631943359 % _timestamp‚1.7845275808635466e+09 _step‚221 0loss/FRt1/assignment_loss‚0.8960223170950656 /loss/FRt2/assignment_loss‚2.046106957229408 2loss/FBt1/assignment_loss‚0.051024476031579495 trainer/global_step‚2699 _runtime‚2545.131455278 2loss/FBt2/assignment_loss‚0.035277135901671204 /loss/FRt2/detection_loss‚0.5934273079230539 1loss/SRqqt1/detection_loss‚0.6402841258349817 /loss/FBt1/detection_loss‚0.8028006583502458Áÿ‹ª ˆ 2loss/SRqqt1/assignment_loss‚0.5954954509916938 1loss/SRqqt2/detection_loss‚0.9881195631943359 % _timestamp‚1.7845275808635466e+09 _step‚221 0loss/FRt1/assignment_loss‚0.8960223170950656 /loss/FRt2/assignment_loss‚2.046106957229408 2loss/FBt1/assignment_loss‚0.051024476031579495 trainer/global_step‚2699 _runtime‚2545.131455278 2loss/FBt2/assignment_loss‚0.035277135901671204 /loss/FRt2/detection_loss‚0.5934273079230539 1loss/SRqqt1/detection_loss‚0.6402841258349817 /loss/FBt1/detection_loss‚0.8028006583502458 &loss/total_loss‚10.078727163641675 epoch‚6 2loss/SRqqt2/assignment_loss‚0.7130339602645588 /loss/FRt1/detection_loss‚1.4212562178983903 /loss/FBt2/detection_loss‚1.2958789929266898ݤ7!ꎫ ‚(:ƒ áõöÒš±¾Þ' disk.nvme1c1n1.out‚8999.97900390625# proc.memory.rssMB‚ 2904.44921875* proc.memory.percent‚17.727351188659668 cpu‚25.67921952151545 network.recv‚680025) disk./.usagePercent‚76.42676883480716$ disk./.usageGB‚1173.341968536377 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚19745.4560546875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.45703125 network.sent‚ 4.23349e+06 disk.md0.in‚0' disk.nvme0c0n1.out‚14440.2197265625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0( proc.memory.availableMB‚ 492.70703125 disk.md0.out‚ 69.18359375 disk.nvme0c0n1.in‚7.1875% disk.nvme1n1.out‚8999.97900390625$ memory_percent‚96.99275493621826 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.dm-1.out‚3546.46533203125% disk.nvme0n1.out‚14440.2197265625²[¬ ‚(:’ áõöÒ‘’ß gpu.0.gpu‚23.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö1tñ’­ ‚(:‡ ðõöÒо›Þ disk.nvme1n1.in‚ 1.45703125# disk.dm-0.out‚19752.52978515625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 proc.cpu.threads‚17) disk./.usagePercent‚76.24979148060933 disk.dm-0.in‚ 8.64453125( disk.nvme0c0n1.out‚14446.86376953125 disk.nvme0n1.in‚7.1875# proc.memory.rssMB‚ 2904.44921875 cpu‚25.678086924713785 network.sent‚ 4.285133e+06% disk.nvme1n1.out‚9001.37060546875 disk.rbd0.in‚0' disk.nvme1c1n1.out‚9001.37060546875 disk.rbd1.out‚0& proc.memory.availableMB‚ 492.453125* proc.memory.percent‚17.727351188659668 network.recv‚684181 disk.md0.in‚0 disk.md0.out‚ 69.54296875" disk.dm-1.out‚3546.63818359375 disk.nvme0c0n1.in‚7.1875& disk.nvme0n1.out‚14446.86376953125 disk.rbd1.in‚0$ memory_percent‚96.99430465698242% disk./.usageGB‚1170.6249237060547µÕCž® ‚(:“ ðõöÒôüæÞ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚270.886' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.29533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R#÷î¯ ‚(:„ ÿõöÒúù€Þ% disk./.usageGB‚1170.5784645080566 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125# proc.memory.rssMB‚ 2904.44921875 network.recv‚687680 disk.nvme0c0n1.in‚7.1875& disk.nvme0n1.out‚14934.78857421875 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚9381.10009765625 cpu‚25.676966073034013) disk./.usagePercent‚76.24676531562004# disk.dm-0.out‚20619.74853515625" disk.dm-1.out‚3546.66943359375( disk.nvme0c0n1.out‚14934.78857421875 disk.nvme0n1.in‚7.1875' disk.nvme1c1n1.out‚9381.10009765625$ memory_percent‚96.99430465698242* proc.memory.percent‚17.727351188659668 network.sent‚ 4.290048e+06 disk.md0.in‚0 disk.md0.out‚69.71875 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 492.453125 proc.cpu.threads‚17‚õ}«ž° ‚(:“ ÿõöÒ¢»òÝ gpu.0.gpu‚75.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.265' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.08833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷§2± - _wandbruntime‚2589  _runtime‚2589HSzõƲ À trainer/global_step‚2749 % _timestamp‚1.7845276254108937e+09 _runtime‚ 2589.67899262 _step‚222 & lr-AdamW/pg1‚0.0001725189179921916 & lr-AdamW/pg2‚0.0001725189179921916ý°ä}˳ Å & lr-AdamW/pg1‚0.0001725189179921916 & lr-AdamW/pg2‚0.0001725189179921916 trainer/global_step‚2749 % _timestamp‚1.7845276254108937e+09 _runtime‚ 2589.67899262 _step‚222Þkœ Æ2´ - _wandbruntime‚2589  _runtime‚2589“ŒÌòá'µ ÛO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420171914534426 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6568330 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3178947267072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416997434731920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22171501 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3174139241232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚422623824927744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚422623466464256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6568057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3179288691200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27152196 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27142053 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3197626656768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4798812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚422623466464256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91509444666880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419444519197184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420171556933792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419444536236544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94707193107968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22171501 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27142053 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4798809 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 _runtime‚2589.683627528 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3174139241232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚422623824927744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚422623466464256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6568057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3179288691200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27152196 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27142053 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3197626656768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4798812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start†“Å3¯/allocated_bytes.all.freed‚422623466464256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91509444666880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419444519197184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420171556933792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419444536236544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94707193107968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22171501 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27142053 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4798809 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 _runtime‚2589.683627528 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6568330ß 4‹2· - _wandbruntime‚2590  _runtime‚2590ïê&¸ äM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4800546 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417144865399392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27161866 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22179294 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11370854 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49341162 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417144848360032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27161866 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27151709 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91542624936960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3198842614784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419592810738688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚422772851065344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419592793699328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22179296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚422773209536000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419592810738688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420320453590262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3180057366016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 trainer/global_step‚2749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22179296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6570581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49341162 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49331003 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27151709 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91542643549184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3175247629484 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚422773209536000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94741486163968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3180398797312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3198741547008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11371127 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420320095989516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344?‚Ïï&¹ éM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚422772851065344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419592793699328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22179296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚422773209536000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419592810738688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420320453590262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3180057366016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 trainer/global_step‚2749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22179296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6570581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49341162 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49331003 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27151709 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91542643549184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3175247629484 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚422773209536000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94741486163968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3180398797312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3198741547008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11371127 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420320095989516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4800546 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417144865399392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27161866 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22179294 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11370854 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49341162 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417144848360032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27161866 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27151709 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91542624936960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3198842614784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419592810738688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496àøxõ2º - _wandbruntime‚2590  _runtime‚2590©Ä» þ 2loss/SRqqt1/assignment_loss‚0.6098794269564264 2loss/SRqqt2/assignment_loss‚0.6124787225435566 /loss/FRt2/detection_loss‚0.5614570535734411 1loss/SRqqt1/detection_loss‚0.6766916997835887 % _timestamp‚1.7845276260415976e+09 /loss/FRt2/assignment_loss‚2.080716244210588 1loss/SRqqt2/detection_loss‚0.9594063181675186 .loss/FBt2/detection_loss‚1.001036389847286 trainer/global_step‚2749 0loss/FRt1/assignment_loss‚0.8679443869927916 .loss/FBt1/detection_loss‚0.696063279110358 %loss/total_loss‚9.645040612756304 _runtime‚2590.312556512 _step‚225 1loss/FBt1/assignment_loss‚0.03155657720155282 2loss/FBt2/assignment_loss‚0.032582896949842954 .loss/FRt1/detection_loss‚1.515227617419353 epoch‚6!W¤ ¼ ƒ _step‚225 1loss/FBt1/assignment_loss‚0.03155657720155282 2loss/FBt2/assignment_loss‚0.032582896949842954 .loss/FRt1/detection_loss‚1.515227617419353 epoch‚6 2loss/SRqqt1/assignment_loss‚0.6098794269564264 2loss/SRqqt2/assignment_loss‚0.6124787225435566 /loss/FRt2/detection_loss‚0.5614570535734411 1loss/SRqqt1/detection_loss‚0.6766916997835887 % _timestamp‚1.7845276260415976e+09 /loss/FRt2/assignment_loss‚2.080716244210588 1loss/SRqqt2/detection_loss‚0.9594063181675186 .loss/FBt2/detection_loss‚1.001036389847286 trainer/global_step‚2749 0loss/FRt1/assignment_loss‚0.8679443869927916 .loss/FBt1/detection_loss‚0.696063279110358 %loss/total_loss‚9.645040612756304 _runtime‚2590.312556512á§y-ƒ½ ‚(:ø ŽööÒ©ñ¶Þ* proc.memory.percent‚17.727351188659668 proc.cpu.threads‚17% disk./.usageGB‚1170.5784301757812 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚9382.3056640625 disk.md0.in‚0 disk.md0.out‚ 70.171875# memory_percent‚96.9930648803711% proc.memory.availableMB‚ 492.65625 network.sent‚ 4.30592e+06 network.recv‚691120" disk.dm-0.out‚20626.5751953125& disk.nvme0c0n1.out‚14941.658203125$ disk.nvme0n1.out‚14941.658203125 disk.nvme1c1n1.in‚ 1.45703125 cpu‚25.675940865831933 disk.dm-1.out‚ 3546.94140625 disk.nvme0c0n1.in‚7.1875 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125# proc.memory.rssMB‚ 2904.44921875 disk.rbd0.in‚0) disk./.usagePercent‚76.24676307935393 disk.dm-0.in‚ 8.64453125 disk.nvme0n1.in‚7.1875 disk.rbd1.out‚0$ disk.nvme1n1.out‚9382.3056640625þï\‹ž¾ ‚(:“ ŽööÒ…ÓôÞ gpu.0.gpu‚85.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.805' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.60166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ +‚¿ ‚(:÷ ööÒ¾’•Þ% disk./.usageGB‚1170.5789413452148 disk.md0.out‚ 70.52734375 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd1.out‚0# memory_percent‚96.9961404800415( proc.memory.availableMB‚ 492.15234375 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 3547.0234375' disk.nvme0c0n1.out‚14942.8798828125" disk.nvme1c1n1.out‚ 9382.859375 disk.nvme1n1.out‚ 9382.859375 network.sent‚ 4.355453e+06 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚20627.4970703125 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2904.44921875* proc.memory.percent‚17.727351188659668 cpu‚25.674655871776846) disk./.usagePercent‚76.24679637487151% disk.nvme0n1.out‚14942.8798828125 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 proc.cpu.threads‚17 network.recv‚695712(ûZÜžÀ ‚(:“ ööÒÒÿ—Þ gpu.0.gpu‚90.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚272.813' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.93766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄXý‹Á ‚(:€ ¬ööÒÅ—ÂÞ disk.dm-1.out‚3547.216796875( disk.nvme0c0n1.out‚14994.47802734375* proc.memory.percent‚17.727375030517578 network.sent‚ 4.362351e+06 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1c1n1.out‚9385.587890625 disk.rbd1.out‚0$ memory_percent‚96.99313640594482 cpu‚25.673562227206677) disk./.usagePercent‚76.36893153908858# disk.nvme1n1.out‚9385.587890625 disk.md0.out‚ 71.2109375 disk.nvme0c0n1.in‚7.1875& disk.nvme0n1.out‚14994.47802734375 disk.rbd0.in‚0( proc.memory.availableMB‚ 492.64453125% disk./.usageGB‚1172.4540214538574 disk.rbd1.in‚0# disk.dm-0.out‚20680.19287109375 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0! proc.memory.rssMB‚ 2904.453125 proc.cpu.threads‚17 network.recv‚698922 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 disk.dm-0.in‚ 8.644531255\Nà ‚(:’ ¬ööÒ¢ãœß gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚137.911' gpu.0.enforcedPY9N½owerLimitWatts‚300.0) gpu.0.powerPercent‚45.970333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06}ï2à - _wandbruntime‚2634  _runtime‚2634•Ok¡ÉÄ Ã _runtime‚2634.887882067 _step‚226 ' lr-AdamW/pg1‚0.00017225284823836092 ' lr-AdamW/pg2‚0.00017225284823836092 trainer/global_step‚2799 % _timestamp‚1.7845276706197767e+09÷~’ÎÅ È ' lr-AdamW/pg1‚0.00017225284823836092 ' lr-AdamW/pg2‚0.00017225284823836092 trainer/global_step‚2799 % _timestamp‚1.7845276706197767e+09 _runtime‚2634.887882067 _step‚226âªæÖ02Æ - _wandbruntime‚2634  _runtime‚2634n\á~à'Ç ÚO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚426858261343744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96421923883008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6682342 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚426858244304384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430092760673280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427598573520800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27636848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3234857793024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 $ _timestamp‚1.784527670620851e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3253465712128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚430093119136768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4885509 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚424368968105520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3229622454640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3234857793024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50187856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50198001 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚426858244304384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22561151 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27626705 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚2634.888997659 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93168458170880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3229963015914 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚2799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6682624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93168476783104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3234516368896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4885512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11568136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚424368951066160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚227 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3234516368896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430092760673280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27626705 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚424368968105520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3229622454640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3234857793024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50187856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50198001 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚426858244304384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22561151 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27626705 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚2634.888997659 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93168458170880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3229963015914 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚2799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6682624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93168476783104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3234516368896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4885512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11568136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚424368951066160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚227 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3234516368896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430092760673280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27626705 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96421923883008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6682342 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚426858244304384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430092760673280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427598573520800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27636848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3234857793024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 $ _timestamp‚1.784527670620851e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3253465712128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452ãŸA;™2É - _wandbruntime‚2635  _runtime‚2635`›æ¹ê&Ê äM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50205305 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3235967899136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4887246 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3235967899136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22568946 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430242503745024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427006535845888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 $ _timestamp‚1.784527671253824e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430242503745024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93201638440960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3230730842892 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11571825 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50215464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3231071404278 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430242145274368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3254688158208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427747470177270 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50215464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427006518806528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27646518 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6684864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚424516381733632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6684582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50205305 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96456345211392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427747112576524 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427006535845888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96456210851328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚2635.522101496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚424516398772992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22568946 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22568944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22568944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3254572410368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272Rm™•!Ë éM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚424516398772992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22568946 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22568944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22568944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3254572410368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50205305 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3235967899136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4887246 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 5ÂIÁÎ 70425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3235967899136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22568946 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430242503745024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427006535845888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 $ _timestamp‚1.784527671253824e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430242503745024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93201638440960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3230730842892 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11571825 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50215464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3231071404278 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430242145274368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3254688158208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427747470177270 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50215464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427006518806528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27646518 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6684864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚424516381733632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6684582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50205305 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96456345211392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427747112576524 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427006535845888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96456210851328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚2635.522101496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656äØF2Ì - _wandbruntime‚2635  _runtime‚2635x\ΣÍ ú 2loss/SRqqt2/assignment_loss‚0.6167209194261142 /loss/SRqqt1/detection_loss‚0.64545260697164 /loss/FBt2/detection_loss‚1.1382302285143824 epoch‚6 /loss/FRt1/assignment_loss‚0.925766905713262 0loss/FRt2/assignment_loss‚2.0511727696324624 0loss/FBt1/assignment_loss‚0.0437922992478842 /loss/FRt2/detection_loss‚0.5716623935542666 1loss/SRqqt2/detection_loss‚1.0684722061589322 _runtime‚2635.522716037 _step‚229 2loss/SRqqt1/assignment_loss‚0.5768613340515374 %loss/total_loss‚9.587455657728935 trainer/global_step‚2799 $ _timestamp‚1.784527671254876e+09 1loss/FBt2/assignment_loss‚0.19097296991608462 -loss/FRt1/detection_loss‚1.31340640715401 /loss/FBt1/detection_loss‚0.4449446173883589qa(GÎ ÿ 2loss/SRqqt2/assignment_loss‚0.6167209194261142 /loss/SRqqt1/detection_loss‚0.64545260697164 /loss/FBt2/detection_loss‚1.1382302285143824 epoch‚6 /loss/FRt1/assignment_loss‚0.925766905713262 0loss/FRt2/assignment_loss‚2.0511727696324624 0loss/FBt1/assignment_loss‚0.0437922992478842 /loss/FRt2/detection_loss‚0.5716623935542666 1loss/SRqqt2/detection_loss‚1.0684722061589322 _runtime‚2635.522716037 _step‚229 2loss/SRqqt1/assignment_loss‚0.5768613340515374 %loss/total_loss‚9.587455657728935 trainer/global_step‚2799 $ _timestamp‚1.784527671254876e+09 1loss/FBt2/assignment_loss‚0.19097296991608462 -loss/FRt1/detection_loss‚1.31340640715401 /loss/FBt1/detection_loss‚0.4449446173883589åÀH&Ï ‚(:‚ »ööÒô†¦Þ& disk.nvme1c1n1.out‚9386.0380859375$ disk.nvme1n1.out‚9386.0380859375 network.recv‚701731# disk.dm-0.out‚22558.78662109375 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.70329475402832 proc.cpu.threads‚17) disk./.usagePercent‚76.42675988974274& disk.nvme0n1.out‚16873.25244140625 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚96.96943759918213( proc.memory.availableMB‚ 496.52734375 cpu‚25.676445259414976 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2900.5078125 network.sent‚ 4.377951e+06% disk./.usageGB‚1173.3418312072754 disk.md0.in‚0 disk.md0.out‚ 71.4609375 disk.dm-1.out‚3547.291015625( disk.nvme0c0n1.out‚16873.25244140625<.Dü“Ð ‚(:ˆ »ööÒ›„æÞ gpu.0.gpu‚93.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.786' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œa$2Ñ - _wandbruntime‚2652  _runtime‚2652•DÇè=Ò â{ @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982181787490845 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27620455622673035 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8031120300292969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 trainer/global_step‚2799 _runtime‚2652.391098935 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09311732649803162 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04408590495586395 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03520365059375763 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6463537812232971 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6680694222450256 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8567107319831848 *jet/accuracy_2_of_4‚0.8248482942581177 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6479449272155762 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4572044610977173 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 jet/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9065831303596497 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3218332827091217 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7039198279380798 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8355631232261658 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04310602694749832 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894960522651672 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8056311011314392 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.760219156742096 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2046964019536972 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2182709127664566 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954005479812622 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8779158592224121 )loss/val_total_loss‚9.195226359090716 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7254270911216736 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2017824947834015 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26679834723472595 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894745349884033 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6538138389587402 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7471903562545776 particle/accuracy_1_of_1‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04814201965928078 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8029595613479614 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38667425513267517 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7544156312942505 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25603678822517395 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2576834261417389 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20856107771396637 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957001209259033 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2756970226764679 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7512669563293457 +jet/accuracy_4_of_4‚0.01628045178949833 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8589599132537842 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6343865990638733 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.810015082359314 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7601237297058105 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8567107319831848 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7862094640731812 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3417626917362213 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03799733519554138 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6983805894851685 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21511423587799072 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5451719164848328 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26590415835380554 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22057922184467316 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.553084671497345 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.470715194940567 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2268846333026886 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6510759592056274 +jet/accuracy_1_of_1‚0.34919455647468567 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22597543895244598 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879391193389893 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6772574186325073 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45477938652038574 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46001678705215454 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21658563613891602 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20263151824474335 +jet/accuracy_3_of_3‚0.06899461150169373 *jet/accuracy_1_of_4‚0.9674410223960876 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27591338753700256 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901585578918457 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6197980642318726 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.573944628238678 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7571621537208557 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958570599555969 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9789921045303345 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_4_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959024786949158 *jet/accuracy_1_of_3‚0.9170141816139221 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7244123220443726 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03799733519554138 /particle/accuracy_4_of_4‚0.9605392217636108 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944067597389221 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7532668113708496 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.845445990562439 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_1_of_5‚1 _step‚230 'particle/f_score‚0.6469201445579529 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4254893362522125 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5467928647994995 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7384164929389954 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06956019997596741 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 (particle/accuracy‚0.8496442437171936 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3082873225212097 /particle/accuracy_3_of_4‚0.9936224818229675 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5972400903701782 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2729400396347046 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03834656998515129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +particle/sensitivity‚0.6414286494255066 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7319391965866089 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987956285476685 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20268292725086212 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04272487387061119 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6868845820426941 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6960774660110474 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.819085419178009 7validation_average_jet_accuracy‚0.38569432497024536 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4264983534812927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.029032429680228233 particle/accuracy_2_of_5‚1 epoch‚6 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6538138389587402 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964741468429565 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6758865714073181 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6910504102706909 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20998360216617584 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_4‚0.08650205284357071 % _timestamp‚1.7845276881224473e+09 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3877235949039459 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7488118410110474 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 *jet/accuracy_2_of_2‚0.1509246826171875 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.551430881023407 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9217928647994995 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27342426776885986 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1253483146429062 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04305053502321243 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5101919770240784 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328705668449402 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6627410054206848 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5305744409561157 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.057428207248449326 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10931839048862457 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6573917865753174 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26679834723472595 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.871397852897644 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10858273506164551 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5091872215270996 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5697026252746582 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871Ðeö‚Ó ç{ 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9065831303596497 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3218332827091217 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7039198279380798 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8355631232261658 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04310602694749832 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894960522651672 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8056311011314392 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.760219156742096 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2046964019536972 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2182709127664566 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954005479812622 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8779158592224121 )loss/val_total_loss‚9.195226359090716 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7254270911216736 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2017824947834015 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26679834723472595 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894745349884033 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6538138389587402 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7471903562545776 particle/accuracy_1_of_1‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04814201965928078 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8029595613479614 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38667425513267517 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7544156312942505 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25603678822517395 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2576834261417389 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20856107771396637 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957001209259033 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2756970226764679 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7512669563293457 +jet/accuracy_4_of_4‚0.01628045178949833 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8589599132537842 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6343865990638733 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.810015082359314 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7601237297058105 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8567107319831848 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7862094640731812 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3417626917362213 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03799733519554138 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6983805894851685 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21511423587799072 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 ^§y2k(Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5451719164848328 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26590415835380554 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22057922184467316 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.553084671497345 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.470715194940567 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2268846333026886 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6510759592056274 +jet/accuracy_1_of_1‚0.34919455647468567 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22597543895244598 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879391193389893 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6772574186325073 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45477938652038574 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46001678705215454 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21658563613891602 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20263151824474335 +jet/accuracy_3_of_3‚0.06899461150169373 *jet/accuracy_1_of_4‚0.9674410223960876 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27591338753700256 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901585578918457 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6197980642318726 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.573944628238678 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7571621537208557 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958570599555969 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9789921045303345 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_4_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959024786949158 *jet/accuracy_1_of_3‚0.9170141816139221 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7244123220443726 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03799733519554138 /particle/accuracy_4_of_4‚0.9605392217636108 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944067597389221 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7532668113708496 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.845445990562439 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_1_of_5‚1 _step‚230 'particle/f_score‚0.6469201445579529 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4254893362522125 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5467928647994995 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7384164929389954 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06956019997596741 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 (particle/accuracy‚0.8496442437171936 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3082873225212097 /particle/accuracy_3_of_4‚0.9936224818229675 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5972400903701782 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2729400396347046 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03834656998515129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +particle/sensitivity‚0.6414286494255066 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7319391965866089 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987956285476685 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20268292725086212 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04272487387061119 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6868845820426941 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6960774660110474 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.819085419178009 7validation_average_jet_accuracy‚0.38569432497024536 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4264983534812927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.029032429680228233 particle/accuracy_2_of_5‚1 epoch‚6 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6538138389587402 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964741468429565 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6758865714073181 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6910504102706909 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20998360216617584 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_4‚0.08650205284357071 % _timestamp‚1.7845276881224473e+09 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3877235949039459 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7488118410110474 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 *jet/accuracy_2_of_2‚0.1509246826171875 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.551430881023407 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9217928647994995 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27342426776885986 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1253483146429062 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04305053502321243 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5101919770240784 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328705668449402 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6627410054206848 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5305744409561157 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.057428207248449326 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10931839048862457 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6573917865753174 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26679834723472595 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.871397852897644 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10858273506164551 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5091872215270996 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5697026252746582 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982181787490845 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27620455622673035 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8031120300292969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 trainer/global_step‚2799 _runtime‚2652.391098935 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09311732649803162 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04408590495586395 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03520365059375763 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6463537812232971 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6680694222450256 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8567107319831848 *jet/accuracy_2_of_4‚0.8248482942581177 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6479449272155762 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4572044610977173 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 jet/accuracy_2_of_5‚1æ´J2{‹Ô ‚(:€ ÊööÒÓº·Þ" disk.dm-0.out‚22562.4345703125 disk.nvme0n1.in‚7.1875% disk.nvme0n1.out‚16876.8193359375& disk.nvme1c1n1.out‚9631.9794921875 disk.rbd1.in‚0$ disk.nvme1n1.out‚9631.9794921875" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 network.sent‚ 4.439513e+06 network.recv‚705678% disk./.usageGB‚1173.3419609069824 disk.dm-1.in‚0! disk.dm-1.out‚3792.7666015625 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚95.49257755279541( proc.memory.availableMB‚ 738.49609375 disk.md0.in‚0 disk.md0.out‚ 71.6328125 disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875' disk.nvme0c0n1.out‚16876.8193359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0) disk./.usagePercent‚76.42676833785914 disk.nvme1n1.in‚ 1.45703125 cpu‚25.686681963391404 proc.cpu.threads‚17EÇa‘žÕ ‚(:“ ÊööÒÅúÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.692' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.89733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;©ÐÂŽÖ ‚(:ƒ ÙööÒû©šÞ disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125( proc.memory.availableMB‚ 738.49609375 network.recv‚709916 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832) disk./.usagePercent‚76.42369297501553 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1n1.out‚9805.23583984375 proc.cpu.threads‚17% disk./.usageGB‚1173.2947463989258 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚17619.12255859375& disk.nvme0n1.out‚17619.12255859375 disk.rbd0.in‚0' disk.nvme1c1n1.out‚9805.23583984375$ memory_percent‚95.49257755279541 cpu‚25.68555109380686 network.sent‚ 4.485785e+06 disk.md0.out‚ 71.7734375" disk.dm-0.out‚23429.6025390625! disk.dm-1.out‚3840.8251953125 disk.nvme0n1.in‚7.1875òø¼Až× ‚(:“ ÙööÒï”ÙÞ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚275.087' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.69566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pïö’Ø ‚(:‡ èööÒ¼Ç¥Þ cpu‚25.684426455602548 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0" disk.dm-1.out‚4038.26904296875( proc.memory.availableMB‚ 738.48828125) proc.memory.percent‚17.70329475402832 proc.cpu.threads‚17 network.sent‚ 4.488707e+06% disk./.usageGB‚1173.2948188781738 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚17632.86669921875 network.recv‚713133) disk./.usagePercent‚76.42369769602175 disk.rbd1.in‚0 disk.md0.out‚ 72.234375( disk.nvme0c0n1.out‚17632.86669921875 disk.rbd0.out‚0$ memory_percent‚95.49262523651123( disk.nvme1c1n1.out‚10004.20361328125 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚10004.20361328125# disk.dm-0.out‚23443.88525390625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2900.5078125v!¤VœÙ ‚(:‘ èööÒœ¿„ß gpu.0.gpu‚70.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚285.7' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.23333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p$¸2Ú - _wandbruntime‚2698  _runtime‚2698ºpêÇÛ Á % _timestamp‚1.7845277337857852e+09 _runtime‚2698.054750005 _step‚231 & lr-AdamW/pg1‚0.0001719814563714146 & lr-AdamW/pg2‚0.0001719814563714146 trainer/global_step‚2849ïpÌÜ Æ & lr-AdamW/pg1‚0.0001719814563714146 & lr-AdamW/pg2‚0.0001719814563714146 trainer/global_step‚2849 % _timestamp‚1.7845277337857852e+09 _runtime‚2698.054750005 _step‚231çü±N¢2Ý - _wandbruntime‚2698  _runtime‚2698wIYà'Þ ÚO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3310403309056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3330028380672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3329929402880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98180097330176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439525075059712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436214313287168 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439524716596224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11812086 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3305409516950 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28192856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94850050337280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439524716596224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436214313287168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23073611 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23073609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0̆–àß ßO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436214313287168 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439524716596224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11812086 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3305409516950 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28192856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94850050337280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439524716596224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436214313287168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocaóë¼# tion.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23073611 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23073609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3310403309056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3330028380672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3329929402880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98180097330176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439525075059712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0èh,Ó¼2à - _wandbruntime‚2699  _runtime‚2699jlDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51294073 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2699.023858923 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4978059 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23081402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94883230607360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28212669 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3331145745920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11816048 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98214275285504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51294073 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28212669 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98214394965504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3306858466588 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3311513408000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433809188154816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439674101197312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4978062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437116046621404 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433809171115456 trainer/global_step‚2849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386·˜ ïð&â êM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51283914 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439674101197312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4978062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437116046621404 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433809171115456 trainer/global_step‚2849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51294073 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2699.023858923 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4978059 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23081402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94883230607360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28212669 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3331145745920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11816048 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98214275285504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51294073 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28212669 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98214394965504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3306858466588 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3311513408000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433809188154816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96509082453504 trainer/global_step‚2899 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443628038394368 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99894944493056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚444394524152576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96úa¢S509063841280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441033290860944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23463261 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446994369268736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23463261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3385862039552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441033273821584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28687651 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6950650 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28677508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446994010805248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3385748381696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.784527779063617e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3365972410880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446994369268736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272inrå'ñ ßO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96509063841280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441033290860944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23463261 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446994369268736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23463261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3385862039552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441033273821584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28687651 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6950650 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28677508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446994010805248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3385748381696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.784527779063617e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3365972410880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446994369268736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443628055433728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚444394166551942 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2743.331830624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52140767 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5063025 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99894812222976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28687651 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚446994010805248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23463259 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3361233291632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12013396 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3366313835008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23463259 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6950371 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3360892730358 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443628055433728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96509082453504 trainer/global_step‚2899 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443628038394368 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99894944493056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚444394524152576ì»ø]2ò - _wandbruntime‚2743  _runtime‚2743ýUóÈë&ó åM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52168375 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845277796925082e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3386861740032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443776312896512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23471052 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99929233920000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447143395406336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6952941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96542262723584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96542244111360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3367082509824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3367423941120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚2899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28697321 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12017703 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28697321 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441180721528416 _runtime‚2743.960835704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3362001118610 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12017419 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23471052 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6952660 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52168375 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443776329935872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚237 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52158216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3386971196416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28687164 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28687164 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441180704489056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52158216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447143753876992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508YûÀô êM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52168375 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845277796925082e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3386861740032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443776312896512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23471052 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99929233920000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447143395406336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6952941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96542262723584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96542244111360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3367082509824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3367423941120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚2899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28697321 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12017703 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2869732×01 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441180721528416 _runtime‚2743.960835704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3362001118610 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12017419 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23471052 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6952660 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52168375 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443776329935872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚237 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52158216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3386971196416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28687164 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28687164 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441180704489056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52158216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447143753876992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452í@Þ’G2õ - _wandbruntime‚2743  _runtime‚2743µ/ö ‚ 0loss/FRt1/assignment_loss‚0.9372520999113718 2loss/FBt2/assignment_loss‚0.005427905008094568 1loss/SRqqt1/detection_loss‚0.6690874636990676 0loss/FRt2/assignment_loss‚1.8893762795894116 2loss/SRqqt2/assignment_loss‚0.6307427904287959 1loss/FBt1/assignment_loss‚0.04521904292961746 /loss/FRt1/detection_loss‚1.5471382048612254 /loss/FRt2/detection_loss‚0.5696729925940409 1loss/SRqqt2/detection_loss‚0.9311017545258414 0loss/FBt1/detection_loss‚0.49627535654881955 .loss/FBt2/detection_loss‚1.216451498646555 2loss/SRqqt1/assignment_loss‚0.5561133036983362 epoch‚7 trainer/global_step‚2899 %loss/total_loss‚9.493858692441178 % _timestamp‚1.7845277796936555e+09 _runtime‚2743.961588766 _step‚238}øä ÷ ‡ 2loss/SRqqt1/assignment_loss‚0.5561133036983362 epoch‚7 trainer/global_step‚2899 %loss/total_loss‚9.493858692441178 % _timestamp‚1.7845277796936555e+09 _runtime‚2743.961588766 _step‚238 0loss/FRt1/assignment_loss‚0.9372520999113718 2loss/FBt2/assignment_loss‚0.005427905008094568 1loss/SRqqt1/detection_loss‚0.6690874636990676 0loss/FRt2/assignment_loss‚1.8893762795894116 2loss/SRqqt2/assignment_loss‚0.6307427904287959 1loss/FBt1/assignment_loss‚0.04521904292961746 /loss/FRt1/detection_loss‚1.5471382048612254 /loss/FRt2/detection_loss‚0.5696729925940409 1loss/SRqqt2/detection_loss‚0.9311017545258414 0loss/FBt1/detection_loss‚0.49627535654881955 .loss/FBt2/detection_loss‚1.216451498646555bÉŽø ‚(:ƒ ¤÷öÒúÏ–Þ& disk.nvme0n1.out‚17638.84912109375 disk.nvme1c1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2900.5078125( disk./.usagePercent‚76.4237389427077# disk.dm-0.out‚23449.28662109375 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.49551010131836& proc.memory.availableMB‚ 738.015625) proc.memory.percent‚17.70329475402832 disk.md0.in‚0( disk.nvme0c0n1.out‚17638.84912109375 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0& disk.nvme1n1.out‚10006.84130859375 cpu‚25.67996876349893 disk.dm-1.in‚0" disk.dm-1.out‚4038.63818359375 disk.rbd0.in‚0( disk.nvme1c1n1.out‚10006.84130859375 disk.rbd1.out‚0 proc.cpu.threads‚17 network.sent‚ 4.566117e+06 network.recv‚727970# disk./.usageGB‚1173.29545211792 disk.md0.out‚ 73.54296875 disk.dm-0.in‚ 8.64453125Lô€®žù ‚(:“ ¤÷öÒ¦‹æÞ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚273.199' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.06633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e“)“ú ‚(:ˆ ³÷öÒÛÜøÝ network.recv‚732424% disk./.usageGB‚1173.2953186035156 disk.rbd1.in‚0& disk.nvme1n1.out‚10007.30126953125 disk.nvme1c1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 network.sent‚ 4.628677e+06 disk.md0.in‚0( disk.nvme0c0n1.out‚17640.17041015625 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚10007.30126953125 cpu‚25.67891094537498) disk./.usagePercent‚76.42373024611729 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚23450.49169921875" disk.dm-1.out‚4038.73193359375 disk.nvme0c0n1.in‚7.1875$ memory_percent‚97.00024127960205 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.md0.out‚ 73.76953125 disk.dm-1.in‚0& disk.nvme0n1.out‚17640.17041015625( proc.memory.availableMB‚ 491.48046875 proc.cpu.threads‚17áêXžû ‚(:“ ³÷öÒñÄóÝ gpu.0.gpu‚60.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.33366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý)8z‹ü ‚(:€ Â÷öÒç–Þ disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚10011.60400390625 disk.rbd1.in‚0 network.sent‚ 4.633554e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 8.64453125! disk.dm-0.out‚23454.013671875" disk.dm-1.out‚4042.04248046875 disk.nvme0n1.in‚7.1875% disk.nvme0n1.out‚17643.6533203125$ memory_percent‚97.00026512145996) disk./.usagePercent‚76.42373248238339 disk.rbd1.out‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 491.4765625) proc.memory.percent‚17.70329475402832 cpu‚25.677945074908806 proc.cpu.threads‚17& disk.nvme1n1.out‚10011.60400390625 disk.md0.out‚74.21875 disk.nvme0c0n1.in‚7.1875' disk.nvme0c0n1.out‚17643.6533203125" proc.memory.rssMB‚ 2900.5078125 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 network.recv‚735792$ disk./.usageGB‚1173.295352935791 disk.rbd0.out‚0´ šžý ‚(:“ Â÷öÒ¢üñÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.863' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.28766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äò’2þ - _wandbruntime‚2788  _runtime‚2788ñÉ/Çÿ Á trainer/global_step‚2949 % _timestamp‚1.7845278244118774e+09 _runtime‚2788.680047248 _step‚239 & lr-AdamW/pg1‚0.0001714227763628549 & lr-AdamW/pg2‚0.0001714227763628549Œ:¶öÌ€ Æ trainer/global_step‚2949 % _timestamp‚1.7845278244118774e+09 _runtime‚2788.680047248 _step‚239 & lr-AdamW/pg1‚0.0001714227763628549 & lr-AdamW/pg2‚0.0001714227763628549ïÓi2 - _wandbruntime‚2788  _runtime‚27882¬®<á'‚ ÛO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53025214 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _step‚240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451041780540928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845278244129715e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29162160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101609797520896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5149725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3421541512704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101609688319488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448404807195184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3416716505040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23852911 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3421882936832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23852911 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98168095957504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454463663477760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53015069 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3421541512704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29162160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454463305014272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454463305014272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7065106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451821183138950 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98168077345280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90589184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29172303 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3416375943766 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451041763501568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441610974208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚ 2788.68112702 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12215120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23852909 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23852909 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¬Ã1Væ'ƒ àO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448404807195184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3416716505040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23852911 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3421882936832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23852911 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98168095957504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454463663477760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53015069 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3421541512704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29162160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454463305014272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454463305014272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7065106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451821183138950 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98168077345280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90589184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29172303 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3416375943766 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451041763501568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441610974208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚ 2788.68112702 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12215120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23852909 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23852909 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53025214 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _step‚240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451041780540928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845278244129715e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29162160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101609797520896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5149725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3421541512704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101609688319488ðŸؤ2„ - _wandbruntime‚2789  _runtime‚2789M§…ð• … çM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5151462 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23860702 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23860704 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚101643978392576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451969722194674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7067669 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3442720777216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417484332018 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53032518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101644102266880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23860702 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29171816 PBDeviceStatsMonitor.on_train_batch_en‰eÖxXd/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53042677 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448552254902016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚2789.313318418 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12218862 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845278250449712e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454613048086016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448552237862656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454613048086016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3417824893404 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3422993042944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7067403 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3422651611648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _step‚241 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53032518 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5151459 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451969722194674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7067669 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3442720777216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417484332018 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53032518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101644102266880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23860702 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29171816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53042677 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448552254902016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚2789.313318418 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12218862 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845278250449712e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454613048086016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176ñëxAî2‡ - _wandbruntime‚2789  _runtime‚2789(ntçˆ € %loss/total_loss‚9.192161580479874 trainer/global_step‚2949 0loss/FRt2/assignment_loss‚2.0034959067327653 2loss/SRqqt1/assignment_loss‚0.5528792424301671 2loss/SRqqt2/assignment_loss‚0.5242554471913389 .loss/FRt1/detection_loss‚1.340045064082554 /loss/FRt2/detection_loss‚0.5832689756414915 /loss/FBt2/detection_loss‚1.0612742568331737 % _timestamp‚1.7845278250461864e+09 1loss/FBt1/assignment_loss‚0.04073868760267593 _runtime‚2789.314088529 epoch‚7 0loss/FRt1/assignment_loss‚0.9352043750030654 1loss/FBt2/assignment_loss‚0.04119641214845555 _step‚242 1loss/SRqqt1/detection_loss‚0.6185543367654333 1loss/SRqqt2/detection_loss‚1.0177764037963808 /loss/FBt1/detection_loss‚0.4734724722523724&ýù ‰ … 2loss/SRqqt1/assignment_loss‚0.5528792424301671 2loss/SRqqt2/assignment_loss‚0.5242554471913389 .loss/FRt1/detection_loss‚1.340045064082554 /loss/FRt2/detection_loss‚0.5832689756414915 /loss/FBt2/detection_loss‚1.0612742568331737 % _timestamp‚1.7845278250461864e+09 1loss/FBt1/assignment_loss‚0.04073868760267593 _runtime‚2789.314088529 epoch‚7 0loss/FRt1/assignment_loss‚0.9352043750030654 1loss/FBt2/assignment_loss‚0.04119641214845555 _step‚242 1loss/SRqqt1/detection_loss‚0.6185543367654333 1loss/SRqqt2/detection_loss‚1.0177764037963808 /loss/FBt1/detection_loss‚0.4734724722523724 %loss/total_loss‚9.192161580479874 trainer/global_step‚2949 0loss/FRt2/assignment_loss‚2.0034959067327653ò¸–§‹Š ‚(:€ Ñ÷öÒµ•¾Þ disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875# disk.nvme0n1.out‚17645.07421875 proc.cpu.threads‚17% disk./.usageGB‚1173.2953910827637( disk.nvme1c1n1.out‚10013.21533203125 disk.rbd1.out‚0& disk.nvme1n1.out‚10013.21533203125 disk.md0.in‚0% disk.nvme0c0n1.out‚17645.07421875 disk.rbd0.in‚0' proc.memory.availableMB‚ 491.6953125 network.sent‚ 4.651264e+06 network.recv‚739084 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0" disk.dm-1.out‚4042.87060546875 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0) disk./.usagePercent‚76.42373496712351 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.out‚ 74.88671875" disk.dm-0.out‚23454.8076171875$ memory_percent‚96.99892997741699" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 cpu‚25.67680976860326Ê/ ¸ž‹ ‚(:“ Ñ÷öÒߘƒß gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.631' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.87699999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÓ)ñŒ ‚(:‚ à÷öÒ…ë£Þ disk.md0.in‚0& disk.nvme0n1.out‚17646.66650390625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚10013.9931640625) proc.memory.percent‚17.70329475402832 network.recv‚743580$ disk./.usageGB‚1173.295337677002 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚23456.17919921875 disk.dm-1.in‚0 disk.dm-1.out‚4043.169921875 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 disk.md0.out‚ 75.2109375 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚17646.66650390625 disk.nvme1c1n1.in‚ 1.45703125# memory_percent‚96.9989538192749( proc.memory.availableMB‚ 491.69140625" proc.memory.rssMB‚ 2900.5078125 proc.cpu.threads‚17 network.sent‚ 4.703024e+06' disk.nvme1c1n1.out‚10013.9931640625 disk.nvme1n1.in‚ 1.45703125 cpu‚25.675785087756275) disk./.usagePercent‚76.42373148848735Œœ ‚(:‘ à÷öÒÜøîÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.692' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.89733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EH")‹Ž ‚(:€ ï÷öÒ§ì„Þ disk.md0.out‚ 75.69921875 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 disk.rbd1.out‚0$ disk.nvme1n1.out‚10014.654296875# disk.dm-0.out‚23456.77294921875& disk.nvme0n1.out‚17647.67919921875 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚1173.295394897461 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.dm-1.out‚4043.208984375$ memory_percent‚96.99907302856445 cpu‚25.674683756155467 network.recv‚747105) disk./.usagePercent‚76.42373521559752 disk.nvme1n1.in‚ 1.45703125 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚17647.67919921875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0& proc.memory.availableMB‚ 491.671875 network.sent‚ 4.707884e+06& disk.nvme1c1n1.out‚10014.654296875 disk.md0.in‚0‰ÌÞ“ ‚(:ˆ ï÷öÒ© ÉÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.507' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.169 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n!Š±2 - _wandbruntime‚2833  _runtime‚2833ûë(È‘  ' lr-AdamW/pg1‚0.00017113552410421445 ' lr-AdamW/pg2‚0.00017113552410421445 trainer/global_step‚2999 $ _timestamp‚1.784527869329255e+09 _runtime‚2833.597389904 _step‚2439®æÍ’ Ç ' lr-AdamW/pg1‚0.00017113552410421445 ' lr-AdamW/pg2‚0.00017113552410421445 trainer/global_step‚2999 $ _timestamp‚1.784527869329255e+09 _runtime‚2833.597389904 _step‚243óVû2“ - _wandbruntime‚2833  _runtime‚2833hŽÞ<š#” ÝO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5236425 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2833.598358465 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103324541212672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚244 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461932599223296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53899516 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚2999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458455505648128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458455488608768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99827109461504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5236428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24242559 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3477110614528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24242561 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3472199718448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471859157174 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚461932957686784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚461932957686784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845278693303187e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3477110614528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12416231 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7179803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3497450363392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3477452038656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103324665094144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24242561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461932599223296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99827090849280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3497555632640 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMoni—¹ƒ™Itor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24242559 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3477452038656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459248199725958 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455776340568784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272@êG è'• âO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458455505648128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29656955 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7179516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53889371 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29646812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29646812 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12415941 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24242559 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3477452038656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459248199725958 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455776340568784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5236425 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2833.598358465 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103324541212672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚244 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461932599223296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53899516 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚2999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458455505648128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458455488608768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99827109461504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5236428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24242559 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3477110614528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24242561 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3472199718448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471859157174 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚461932957686784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚461932957686784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845278693303187e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3477110614528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12416231 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7179803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3497450363392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3477452038656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103324665094144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24242561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461932599223296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99827090849280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3497555632640 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419913 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53916979 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24250352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455923788275616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53916979 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3478562144768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24250354 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚2999 _runtime‚2834.230837618 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462082342295040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚245 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3498677191680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24250352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603780150272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459396738781682 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3478220713472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603763110912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103358966923264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459397096382428 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3478220713472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29656468 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7181754 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29666625 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603763110912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462082342295040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458603780150272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472967545426 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462081983824384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24250354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3498563540992 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53906820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99860271119360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3478562144768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29656468 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12420214 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29666625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3473308106812 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53906820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99860289731584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224€©o"ò&˜ ìM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3478220713472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603763110912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103358966923264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459397096382428 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3478220713472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29656468 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7181754 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29666625 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603763110912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462082342295040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458603780150272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472967545426 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462081983824384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24250354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3498563540992 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53906820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99860271119360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3478562144768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29656468 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12420214 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29666625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3473308106812 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53906820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99860289731584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419913 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53916979 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24250352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455923788275616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53916979 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3478562144768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24250354 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚2999 _runtime‚2834.230837618 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462082342295040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚245 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3498677191680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24250352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603780150272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459396738781682 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2õSPà2™ - _wandbruntime‚2834  _runtime‚2834úØ`¬š ‚ trainer/global_step‚2999 0loss/FRt1/assignment_loss‚0.9299572354796305 /loss/FRt1/detection_loss‚1.4156754095311885 /loss/FRt2/detection_loss‚0.6188722090926798 1loss/SRqqt2/detection_loss‚1.0711714152354785 2loss/FBt1/assignment_loss‚0.033543601667915035 % _timestamp‚1.7845278699635482e+09 /loss/FBt1/detection_loss‚0.5159497213441752 epoch‚7 0loss/FRt2/assignment_loss‚1.9618554248775368 2loss/SRqqt1/assignment_loss‚0.6252247031160585 2loss/SRqqt2/assignment_loss‚0.7426950215296851 2loss/FBt2/assignment_loss‚0.019695634716812106 _runtime‚2834.234577523 _step‚246 0loss/SRqqt1/detection_loss‚0.654127313675894 /loss/FBt2/detection_loss‚0.8545012733545797 %loss/total_loss‚9.443268963621634~Ë *L› ‡ /loss/FBt2/detection_loss‚0.8545012733545797 %loss/total_loss‚9.443268963621634 trainer/global_step‚2999 0loss/FRt1/assignment_loss‚0.9299572354796305 /loss/FRt1/detection_loss‚1.4156754095311885 /loss/FRt2/detection_loss‚0.6188722090926798 1loss/SRqqt2/detection_loss‚1.0711714152354785 2loss/FBt1/assignment_loss‚0.033543601667915035 % _timestamp‚1.7845278699635482e+09 /loss/FBt1/detection_loss‚0.5159497213441752 epoch‚7 0loss/FRt2/assignment_loss‚1.9618554248775368 2loss/SRqqt1/assignment_loss‚0.6252247031160585 2loss/S?DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3527342370582 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24632211 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 $ _timestamp‚1.784527914639435e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105039372107776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30131464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5323128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3049 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚465869213715968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469401893432320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3533021140480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24632211 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3553267754496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463147890981744 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24632209 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469401893432320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 119949312 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12617554 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105039510669312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466675573913600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54763673ßI9žé'§ ãO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3532679716352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3553387703808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3527682931856 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469402251895808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7294426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24632209 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚465869230755328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 138561536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101486122965504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466675216312966 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3533021140480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30141607 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚465869230755328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3527342370582 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24632211 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 $ _timestamp‚1.784527914639435e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105039372107776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30131464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5323128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3049 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚465869213715968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469401893432320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3533021140480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24632211 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3553267754496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463147890981744 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24632209 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469401893432320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 119949312 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12617554 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105039510669312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466675573913600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54763673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BŒc© éM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24640004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30151277 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚466017488218112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54781122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463295321649216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30151277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚2879.539889356 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105073799948288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466017488218112 % _timestamp‚1.7845279152711968e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469551636504064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12621533 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3534131246592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24640002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5324862 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5324859 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463295304609856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101519303235584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24640004 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30141120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466824112969436 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54791281 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7296398 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101519284623360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚466823755368690 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12621257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3533789815296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7296671 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466017505257472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24640002 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0’(} ô&ª îM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24640004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30151277 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚466017488218112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54781122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463295321649216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30151277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚2879.539889356 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105073799948288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466017488218112 % _timestamp‚1.7845279152711968e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469551636504064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12621533 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3534131246592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24640002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5324862 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5324859 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463295304609856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101519303235584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24640004 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30141120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466824112969436 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54791281 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7296398 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101519284623360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚466823755368690 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12621257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3533789815296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7296671 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466017505257472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24640002 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ù]Ñé2« - _wandbruntime‚2879  _runtime‚2879€Ž¢¬  %loss/total_loss‚9.253799319760102 2loss/SRqqt1/assignment_loss‚0.5761027018441548 2loss/FBt2/assignment_loss‚0.025836377816428543 1loss/SRqqt2/detection_loss‚0.9711440292614624 2loss/SRqqt2/assignment_loss‚0.5766955207375833 epoch‚7 0loss/FRt1/assignment_loss‚0.8187655122553716 /loss/FRt2/assignment_loss‚1.994439049536748 2loss/FBt1/assignment_loss‚0.005620445960278209 /loss/FRt1/detection_loss‚1.4582136027824104 1loss/SRqqt1/detection_loss‚0.5873734486214578 trainer/global_step‚3049 _step‚250 /loss/FRt2/detection_loss‚0.5500956670928685 /loss/FBt1/detection_loss‚0.6614693143194371 % _timestamp‚1.7845279152726884e+09 _runtime‚2879.540710967 .loss/FBt2/detection_loss‚1.028043649531901èH ­ † 2loss/SRqqt2/assignment_loss‚0.5766955207375833 epoch‚7 0loss/FRt1/assignment_loss‚0.8187655122553716 /loss/FRt2/assignment_loss‚1.994439049536748 2loss/FBt1/assignment_loss‚0.005620445960278209 /loss/FRt1/detection_loss‚1.4582136027824104 1loss/SRqqt1/detection_loss‚0.5873734486214578 trainer/global_step‚3049 _step‚250 /loss/FRt2/detection_loss‚0.5500956670928685 /loss/FBt1/detection_loss‚0.6614693143194371 % _timestamp‚1.7845279152726884e+09 _runtime‚2879.540710967 .loss/FBt2/detection_loss‚1.028043649531901 %loss/total_loss‚9.253799319760102 2loss/SRqqt1/assignment_loss‚0.5761027018441548 2loss/FBt2/assignment_loss‚0.025836377816428543 1loss/SRqqt2/detection_loss‚0.9711440292614624ú3½ÙŠ® ‚(:ÿ «øöÒÝüÝ# disk.dm-0.out‚23461.65087890625 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0& disk.nvme1c1n1.out‚10552.013671875" proc.memory.rssMB‚ 2900.5078125) disk./.usagePercent‚76.42377348059533 disk.md0.in‚0 disk.md0.out‚ 76.97265625! disk.dm-1.out‚4578.5966796875& disk.nvme0n1.out‚17653.34912109375 disk.rbd0.out‚0$ disk.nvme1n1.out‚10552.013671875% proc.memory.availableMB‚ 491.34375 network.recv‚761592% disk./.usageGB‚1173.2959823608398 disk.rbd1.in‚0# memory_percent‚97.0010757446289) proc.memory.percent‚17.70329475402832 cpu‚25.670634722202507 proc.cpu.threads‚17 network.sent‚ 4.78303e+06 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚17653.34912109375 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125×: ‚(:‘ «øöÒŠ£øÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.608' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.86933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±… ‰° ‚(:þ ºøöÒÿþžÞ" proc.memory.rssMB‚ 2900.5078125( disk./.usagePercent‚76.4237898798801 disk.md0.in‚0" disk.dm-0.out‚23464.8466796875 disk.dm-1.in‚0! disk.dm-1.out‚4578.8740234375' disk.nvme0c0n1.out‚17656.4931640625% disk.nvme0n1.out‚17656.4931640625 disk.rbd1.out‚0% disk./.usageGB‚1173.2962341308594 disk.dm-0.in‚ 8.64453125 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.99974060058594) proc.memory.percent‚17.70329475402832 cpu‚25.669750589640216 proc.cpu.threads‚17 network.recv‚765999 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚10552.9658203125$ proc.memory.availableMB‚491.5625 network.sent‚ 4.846604e+06 disk.md0.out‚ 77.25390625 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0% disk.nvme1n1.out‚10552.9658203125‰¹ˆž± ‚(:“ ºøöÒû°æÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚131.393' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.79766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<Ô¼Œ² ‚(: ÉøöÒΘÞ$ memory_percent‚96.99978828430176' proc.memory.availableMB‚ 491.5546875% disk./.usageGB‚1173.2961769104004 disk.dm-1.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0% disk.nvme1n1.out‚10553.3681640625 disk.md0.in‚0" disk.dm-0.out‚23465.6005859375" proc.memory.rssMB‚ 2900.5078125 network.sent‚ 4.851505e+06) disk./.usagePercent‚76.42378615276994 disk.nvme1n1.in‚ 1.45703125 disk.md0.out‚ 77.4765625 disk.dm-0.in‚ 8.64453125) proc.memory.percent‚17.70329475402832 cpu‚25.668796123483837 proc.cpu.threads‚17 network.recv‚769540! disk.dm-1.out‚4578.9130859375% disk.nvme0n1.out‚17657.3681640625' disk.nvme1c1n1.out‚10553.3681640625 disk.nvme0c0n1.in‚7.1875' disk.nvme0c0n1.out‚17657.3681640625 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0Kð¬Ø‘³ ‚(:† ÉøöÒüõåÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.839' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.613 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B¦|ä2´ - _wandbruntime‚2924  _runtime‚2924=Ü,àǵ Á ' lr-AdamW/pg1‚0.00017054528732540644 ' lr-AdamW/pg2‚0.00017054528732540644 trainer/global_step‚3099 # _timestamp‚1.78452796007052e+09 _runtime‚2924.338691542 _step‚251l ¾Ì¶ Æ ' lr-AdamW/pg1‚0.00017054528732540644 ' lr-AdamW/pg2‚0.00017054528732540644 trainer/global_step‚3099 # _timestamp‚1.78452796007052e+09 _runtime‚2924.338691542 _step‚251û6Ñå®2· - _wandbruntime‚2924  _runtime‚2924Ïùà²å'¸ ßO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55637975 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12818543 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845279600715904e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7408715 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚474102590500608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚476871187641344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚476871546104832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚252 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30626259 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7408429 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103145136469504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474102232899974 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25021861 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473282938823168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473282955862528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚476871187641344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3609050044416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚470519407315984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3582825583990 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚476871546104832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103145117857280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3583166145264 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3588248818176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25021861 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30626259 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409828 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3609161605120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30616116 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55648120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3588590242304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180`º€g~¹ äO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55637975 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12818543 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845279600715904e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7408715 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚474102590500608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚476871187641344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚476871546104832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚252 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30626259 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7408429 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103145136469504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474102232899974 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25021861 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473282938823168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473282955862528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚476871187641344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_byË]¦ltes.small_pool.freed‚ 3609050044416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚470519407315984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3582825583990 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚476871546104832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103145117857280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3583166145264 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3588248818176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25021861 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30626259 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409828 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3609161605120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30616116 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55648120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3588590242304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ü´%©Ð2º - _wandbruntime‚2924  _runtime‚2924&âúÀï&» éM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚3099 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473431213325312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3584274533628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845279607035334e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12822253 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚470666837983456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55665583 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55665583 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25029652 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2924.971928462 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473431230364672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473431230364672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12822534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30635929 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477020930713088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5411562 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103178316739584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3610277995520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 35839339722422P¼šô&¼ îM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25029654 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477020572242432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚474250771955698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473431213325312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3584274533628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845279607035334e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12822253 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚470666837983456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55665583 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55665583 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25029652 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2924.971928462 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473431230364672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473431230364672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12822534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30635929 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477020930713088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5411562 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103178316739584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3610277995520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3583933972242 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚3099 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108469151496192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚484340840313856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3643817920000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3644159344128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108469044391936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚484340481850368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3665001522688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25411509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13019940 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481529249486982 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2969.293167268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚480696663930368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25411511 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3664913030656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31110911 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3643817920000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚484340840313856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3644159344128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31110911 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477890940689584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚484340481850368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚3149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25411511 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚477890957728944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7523690 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56512277 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5496528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5496525 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚480696663930368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56522422 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481529607087616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚484340481850368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3665001522688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25411509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13019940 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481529249486982 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2969.293167268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚480696663930368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25411511 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3664913030656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31110911 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3643817920000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚484340840313856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3644159344128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31110911 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477890940689584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚484340481850368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚3149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25411511 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚477890957728944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7523690 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56512277 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5496528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5496525 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚480696663930368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56522422 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481529607087616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108469151496192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0€i3†2Ì - _wandbruntime‚2969  _runtime‚2969:/&žï&Í éM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31110424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7525696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚480844938432512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3666020906496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56529726 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56539885 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13023955 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3644928018944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108503332537856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3666126168576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104837311631360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25419304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56539885 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5498262 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚480844955471872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25419304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108503456412160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚484490224922112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480844938432512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3639757747036 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚484490224922112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚478038388396416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31120581 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚484489866451456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3645269450240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚478038371357056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3644928018944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481678146143452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31120581 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56529726 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25419302 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104837330243584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581¼2|ÞR Î îM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚478038388396416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31120581 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚484489866451456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3645269450240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚478038371357056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3644928018944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481678146143452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31120581 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56529726 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25419302 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104837330243584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31110424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7525696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split-^ Ê¢.small_pool.peak‚508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚480844938432512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3666020906496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56529726 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56539885 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13023955 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3644928018944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108503332537856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3666126168576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104837311631360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25419304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56539885 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5498262 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚480844955471872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25419304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108503456412160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚484490224922112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480844938432512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3639757747036 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚484490224922112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184vªÌ2Ï - _wandbruntime‚2969  _runtime‚2969¿QP4Ð € 2loss/SRqqt2/assignment_loss‚0.6741311678306374 3loss/FBt2/assignment_loss‚0.0007173449093460687 /loss/FRt1/detection_loss‚1.4514402334539531 /loss/FRt2/detection_loss‚0.5617911208291652 1loss/SRqqt2/detection_loss‚0.9795403697094621 .loss/FBt2/detection_loss‚1.410496158955189 _runtime‚2969.926686147 0loss/FRt1/assignment_loss‚0.8761147728810708 0loss/FRt2/assignment_loss‚1.9899034813451624 2loss/SRqqt1/assignment_loss‚0.5036860711239473 /loss/FBt1/detection_loss‚0.5412718897239444 trainer/global_step‚3149 _step‚258 epoch‚7 1loss/SRqqt1/detection_loss‚0.6800292159984043 %loss/total_loss‚9.701265553539946 # _timestamp‚1.78452800565869e+09 1loss/FBt1/assignment_loss‚0.03214372677966335ïùb Ñ … 2loss/SRqqt1/assignment_loss‚0.5036860711239473 /loss/FBt1/detection_loss‚0.5412718897239444 trainer/global_step‚3149 _step‚258 epoch‚7 1loss/SRqqt1/detection_loss‚0.6800292159984043 %loss/total_loss‚9.701265553539946 # _timestamp‚1.78452800565869e+09 1loss/FBt1/assignment_loss‚0.03214372677966335 2loss/SRqqt2/assignment_loss‚0.6741311678306374 3loss/FBt2/assignment_loss‚0.0007173449093460687 /loss/FRt1/detection_loss‚1.4514402334539531 /loss/FRt2/detection_loss‚0.5617911208291652 1loss/SRqqt2/detection_loss‚0.9795403697094621 .loss/FBt2/detection_loss‚1.410496158955189 _runtime‚2969.926686147 0loss/FRt1/assignment_loss‚0.8761147728810708 0loss/FRt2/assignment_loss‚1.9899034813451624‚çKH«ŒÒ ‚(: …ùöÒ£íŒÞ cpu‚25.6648204740618% disk./.usageGB‚1173.2973518371582 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚23471.9521484375' disk.nvme0c0n1.out‚17664.3662109375 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0$ memory_percent‚97.00253009796143) proc.memory.percent‚17.70329475402832 network.sent‚ 4.926624e+06) disk./.usagePercent‚76.42386268276556 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1n1.out‚10556.1650390625 disk.md0.in‚0 disk.md0.out‚ 78.91796875! disk.dm-1.out‚4579.2197265625% disk.nvme0n1.out‚17664.3662109375( proc.memory.availableMB‚ 491.10546875" proc.memory.rssMB‚ 2900.5078125 network.recv‚783233' disk.nvme1c1n1.out‚10556.1650390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875n…ÞžÓ ‚(:“ …ùöÒôðüÞ gpu.0.gpu‚91.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚193.544' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.51466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š´P‡Ô ‚(:ü ”ùöÒЯ„Þ( disk.nvme0c0n1.out‚17667.72216796875& disk.nvme0n1.out‚17667.72216796875% disk.nvme1c1n1.out‚10557.30859375# disk.nvme1n1.out‚10557.30859375 cpu‚25.66391445442652 proc.cpu.threads‚17) disk./.usagePercent‚76.42386939156387 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚23475.07763671875 disk.dm-1.out‚ 4579.4140625 disk.rbd0.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125# memory_percent‚97.0029354095459% disk./.usageGB‚1173.2974548339844 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.md0.in‚0 disk.md0.out‚ 79.46484375' proc.memory.availableMB‚ 491.0390625) proc.memory.percent‚17.70329475402832 network.sent‚ 4.990118e+06 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875" proc.memory.rssMB‚ 2900.5078125 network.recv‚787151 disk.dm-1.in‚0ódí?Õ ‚(:’ ”ùöÒ­–œÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.531' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.843666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íòÌ%†Ö ‚(:û £ùöÒ¹Ù…Þ disk.dm-1.out‚ 4579.4375' disk.nvme0c0n1.out‚17669.1044921875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0$ memory_percent‚97.00310230255127) proc.memory.percent‚17.70329475402832 disk.md0.out‚ 79.83984375 disk.dm-0.in‚ 8.64453125% disk./.usageGB‚1173.2967109680176' disk.nvme1c1n1.out‚10557.8251953125% disk.nvme0n1.out‚17669.1044921875( proc.memory.availableMB‚ 491.01171875" proc.memory.rssMB‚ 2900.5078125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 1.45703125 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0 cpu‚25.66293335826863! disk.dm-0.out‚23476.154296875 network.recv‚790425) disk./.usagePercent‚76.42382093913159 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚10557.8251953125 disk.dm-1.in‚0 network.sent‚ 4.995776e+06 disk.md0.in‚09c³øž× ‚(:“ £ùöÒƃòÝ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.105' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.36833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L"2Ø - _wandbruntime‚3014  _runtime‚3014ªï«ÿÉÙ Ã % _timestamp‚1.7845280502652082e+09 _runtime‚3014.533301265 _step‚259 ' lr-AdamW/pg1‚0.00016993420112270774 ' lr-AdamW/pg2‚0.00016993420112270774 trainer/global_step‚3199y~!&ÎÚ È % _timestamp‚1.7845280502652082e+09 _runtime‚3014.533301265 _step‚259 ' lr-AdamW/pg1‚0.00016993420112270774 ' lr-AdamW/pg2‚0.00016993420112270774 trainer/global_step‚3199ƒõêÕh2Û - _wandbruntime‚3014  _runtime‚3014"]¢ä'Ü ÞO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5583228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106463144865280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3720728603648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57386579 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13221012 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784528050265928e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31595563 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13220724 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488956623674624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485262491102544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25801161 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491810134522880 trainer/global_step‚3199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31585420 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7637784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110184001544704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚3014.533928356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491810134522880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚260 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31585420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106463163477504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3720838067200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31595563 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485262474063184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491809776059392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491809776059392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25801159 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7637499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110183873468928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3699728445952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25801161 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3699387021824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3699387021824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488110406076928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5583225 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3693792010806 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488110406076928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚488956266073990 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3694132572080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488110389037568 4‹Z#Ý ãO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57386579 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13221012 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784528050265928e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31595563 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13220724 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488956623674624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485262491102544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25801161 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491810134522880 trainer/global_step‚3199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31585420 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7637784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110184001544704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚3014.533928356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491810134522880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚260 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31585420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106463163477504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3720838067200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31595563 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485262474063184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491809776059392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491809776059392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25801159 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7637499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110183873468928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3699728445952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25801161 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3699387021824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3699387021824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488110406076928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5583225 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3693792010806 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488110406076928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_tra-'in_batch_start/requested_bytes.all.freed‚488956266073990 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3694132572080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488110389037568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5583228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106463144865280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3720728603648„ØW’…Þ ‚(:ú ²ùöÒŒ ÆÞ# disk.nvme0n1.out‚17671.44140625 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚10558.70458984375" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832( disk./.usagePercent‚76.4238152242293 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚10558.70458984375 disk.rbd1.out‚0 disk.md0.in‚0 disk.md0.out‚ 80.171875 cpu‚25.662118219533777 network.sent‚ 5.011583e+06% disk./.usageGB‚1173.2966232299805" disk.dm-1.out‚4579.62255859375 disk.rbd1.in‚0$ memory_percent‚97.00498580932617 disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875% disk.nvme0c0n1.out‚17671.44140625 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 490.703125 proc.cpu.threads‚17 network.recv‚793654 disk.dm-0.out‚ 23478.4453125 disk.dm-1.in‚0íµM“ß ‚(:ˆ ²ùöÒ•†ÐÞ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.421' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.807 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰™zã2à - _wandbruntime‚3015  _runtime‚3015ïî&á èM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488258680579072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694900399058 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57414187 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5584959 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3721843944448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚3199 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25808954 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485409904730656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚ 3015.16205473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106496325135360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845280508937685e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3700838552064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57414187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13224994 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25808954 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488258680579072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110218290856960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3721947109376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13224731 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5584962 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7639772 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3700838552064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31595076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485409921770016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3700497120768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31605233 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491959519131136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57404028 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491959519131136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488258663539712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57414187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13224994 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25808954 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488258680579072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110218290856960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3721947109376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13224731 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5584962 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7639772 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3700838552064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31595076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485409921770016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3700497120768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31605233 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491959519131136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57404028 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491959519131136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488258663539712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106496325135360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845280508937685e+09…ýîã©2ã - _wandbruntime‚3015  _runtime‚3015põŽˆä € /loss/FRt1/detection_loss‚1.3814455869765292 _runtime‚3015.162843792 0loss/FRt1/assignment_loss‚0.8537135946703708 2loss/SRqqt2/assignment_loss‚0.7154641610359891 /loss/FRt2/detection_loss‚0.5795011209879506 1loss/SRqqt2/detection_loss‚0.9542747892275014 .loss/FBt2/detection_loss‚1.296167363059418 % _timestamp‚1.7845280508948796e+09 1loss/FBt1/assignment_loss‚0.06030000202697479 1loss/FBt2/assignment_loss‚0.02057767692296496 1loss/SRqqt1/detection_loss‚0.6270536912288006 %loss/total_loss‚9.557926151435403 trainer/global_step‚3199 _step‚262 0loss/FRt2/assignment_loss‚2.0100401273511417 2loss/SRqqt1/assignment_loss‚0.5335992697961194 /loss/FBt1/detection_loss‚0.5257887681516427 epoch‚7Ÿ‡ å … % _timestamp‚1.7845280508948796e+09 1loss/FBt1/assignment_loss‚0.06030000202697479 1loss/FBt2/assignment_loss‚0.02057767692296496 1loss/SRqqt1/detection_loss‚0.6270536912288006 %loss/total_loss‚9.557926151435403 trainer/global_step‚3199 _step‚262 0loss/FRt2/assignment_loss‚2.0100401273511417 2loss/SRqqt1/assignment_loss‚0.5335992697961194 /loss/FBt1/detection_loss‚0.5257887681516427 epoch‚7 /loss/FRt1/detection_loss‚1.3814455869765292 _runtime‚3015.162843792 0loss/FRt1/assignment_loss‚0.8537135946703708 2loss/SRqqt2/assignment_loss‚0.7154641610359891 /loss/FRt2/detection_loss‚0.5795011209879506 1loss/SRqqt2/detection_loss‚0.9542747892275014 .loss/FBt2/detection_loss‚1.296167363059418†*“æ ‚(:ˆ ÁùöÒ²Þ network.sent‚ 5.050463e+06 disk.md0.out‚ 80.42578125 disk.dm-0.in‚ 8.64453125 disk.rbd0.in‚0 proc.cpu.threads‚17" disk.dm-1.out‚4582.58349609375 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚17672.58447265625& disk.nvme0n1.out‚17672.58447265625( disk.nvme1c1n1.out‚10562.01904296875( proc.memory.availableMB‚ 490.84765625" proc.memory.rssMB‚ 2900.5078125 cpu‚25.673309074034343# disk.dm-0.out‚23479.37548828125 network.recv‚797631) disk./.usagePercent‚76.42385249533108% disk./.usageGB‚1173.2971954345703& disk.nvme1n1.out‚10562.01904296875 disk.md0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125# memory_percent‚97.0041036605835) proc.memory.percent‚17.70329475402832-›/žç ‚(:“ ÁùöÒ§Í”Þ gpu.0.gpu‚76.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.321' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.77366666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¸ïÝ2è - _wandbruntime‚3031  _runtime‚3031é%Q+Eé ™{ 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9822821021080017 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46732720732688904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8396258354187012 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976717829704285 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8078230619430542 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1108851283788681 *jet/accuracy_1_of_2‚0.7825865149497986 particle/accuracy_3_of_3‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.735401451587677 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938408732414246 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7618213295936584 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9648973345756531 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5983538627624512 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2370327115058899 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9242366552352905 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7validation_average_jet_accuracy‚0.39102625846862793 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8283824920654297 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6945346593856812 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.752262532711029 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24680741131305695 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7210617065429688 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3036981523036957 trainer/global_step‚3199 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2142278105020523 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02434595115482807 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9696753621101379 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4498704671859741 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025443265214562416 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5619491934776306 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7412439584732056 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7677372694015503 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019800644367933273 ,jet/accuracy_4_of_4‚0.013899499550461769 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987694501876831 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11078745871782303 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12718838453292847 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019800644367933273 particle/accuracy_3_of_4‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34729698300361633 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8820815086364746 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5551504492759705 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18289712071418762 *jet/accuracy_1_of_3‚0.9125386476516724 epoch‚7 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950242638587952 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05691678076982498 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8513190150260925 'particle/f_score‚0.6463383436203003 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6567279100418091 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20372025668621063 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23390717804431915 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.35718610882759094 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11499999463558197 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5581493973731995 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6517573595046997 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6525073051452637 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5463804006576538 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04508255794644356 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24426591396331787 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6460479497909546 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22041675448417664 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6344327926635742 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9639344811439514 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.386442631483078 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21025267243385315 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7362245321273804 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5300000309944153 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9936086535453796 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6525073051452637 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/2FRt0SRqù§ÂÇZ!qt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926055073738098 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7440308332443237 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2941347658634186 +particle/specificity‚0.9088020324707031 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5729166269302368 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975327849388123 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4337947964668274 particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9848375916481018 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6954658031463623 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44731202721595764 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22870615124702454 +jet/accuracy_3_of_3‚0.07096494734287262 particle/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863697350025177 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 3031.8448829 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 % _timestamp‚1.7845280675762782e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6996613144874573 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6896240711212158 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9135586023330688 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6761290431022644 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7370020747184753 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.817158043384552 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5085740089416504 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30268728733062744 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6202726364135742 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2858104407787323 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 (particle/accuracy‚0.8503766655921936 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7347241640090942 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10145367681980133 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 *jet/accuracy_2_of_4‚0.7983990907669067 +jet/accuracy_3_of_4‚0.08900831639766693 )loss/val_total_loss‚9.008392249884336 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984030723571777 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6685553789138794 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9843522906303406 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32224100828170776 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083432674408 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4809873700141907 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3841158151626587 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4487251937389374 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6456294655799866 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033580999821424484 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9949015378952026 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8633753657341003 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3030965328216553 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025044267997145653 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2941347658634186 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027122411876916885 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35521936416625977 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975327849388123 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2933580279350281 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37037038803100586 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5113519430160522 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8319728374481201 *particle/sensitivity‚0.636717677116394 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6863327026367188 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30277886986732483 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7290052771568298 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05217171832919121 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20870889723300934 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6777745485305786 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.73525071144104 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2241353988647461 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7524542212486267 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6481791734695435 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8513190150260925 *jet/accuracy_2_of_3‚0.5677697062492371 particle/accuracy_1_of_2‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.098844014108181 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3004119098186493 +jet/accuracy_2_of_2‚0.15226800739765167 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7983889579772949 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8000212907791138 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5750712156295776 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_5_of_5‚0 _step‚263 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2857176959514618 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11420264840126038ÏÐ.¤=ê ž{ ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033580999821424484 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9949015378952026 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8633753657341003 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3030965328216553 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025044267997145653 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2941347658634186 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027122411876916885 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35521936416625977 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975327849388123 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2933580279350281 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37037038803100586 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5113519430160522 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8319728374481201 *particle/sensitivity‚0.636717677116394 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6863327026367188 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30277886986732483 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7290052771568298 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05217171832919121 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20870889723300934 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6777745485305786 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.73525071144104 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2241353988647461 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7524542212486267 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6481791734695435 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8513190150260925 *jet/accuracy_2_of_3‚0.5677697062492371 particle/accuracy_1_of_2‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.098844014108181 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3004119098186493 +jet/accuracy_2_of_2‚0.15226800739765167 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7983889579772949 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8000212907791138 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5750712156295776 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_5_of_5‚0 _step‚263 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2857176959514618 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11420264840126038 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9822821021080017 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46732720732688904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8396258354187012 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976717829704285 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8078230619430542 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1108851283788681 *jet/accuracy_1_of_2‚0.7825865149497986 particle/accuracy_3_of_3‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.735401451587677 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938408732414246 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7618213295936584 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9648973345756531 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5983538627624512 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2370327115058899 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9242366552352905 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7validation_average_jet_accuracy‚0.39102625846862793 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8283824920654297 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6945346593856812 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.752262532711029 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24680741131305695 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7210617065429688 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3036981523036957 trainer/global_step‚3199 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2142278105020523 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02434595115482807 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9696753621101379 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4498704671859741 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025443265214562416 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5619491934776306 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7412439584732056 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7677372694015503 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019800644367933273 ,jet/accuracy_4_of_4‚0.013899499550461769 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987694501876831 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11078745871782303 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12718838453292847 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019800644367933273 particle/accuracy_3_of_4‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34729698300361633 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8820815086364746 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5551504492759705 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18289712071418762 *jet/accuracy_1_of_3‚0.9125386476516724 epoch‚7 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950242638587952 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05691678076982498 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8513190150260925 'particle/f_score‚0.6463383436203003 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6567279100418091 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20372025668621063 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23390717804431915 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.35718610882759094 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11499999463558197 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5581493973731995 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6517573595046997 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6525073051452637 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5463804006576538 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04508255794644356 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24426591396331787 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6460479497909546 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22041675448417664 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6344327926635742 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9639344811439514 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.386442631483078 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21025267243385315 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7362245321273804 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5300000309944153 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9936086535453796 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6525073051452637 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926055073738098 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7440308332443237 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2941347658634186 +particle/specificity‚0.9088020324707031 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5729166269302368 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975327849388123 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4337947964668274 particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9848375916481018 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6954658031463623 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44731202721595764 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22870615124702454 +jet/accuracy_3_of_3‚0.07096494734287262 particle/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863697350025177 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 3031.8448829 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 % _timestamp‚1.7845280675762782e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6996613144874573 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6896240711212158 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9135586023330688 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6761290431022644 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7370020747184753 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.817158043384552 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5085740089416504 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30268728733062744 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6202726364135742 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2858104407787323 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 (particle/accuracy‚0.8503766655921936 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7347241640090942 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10145367681980133 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 *jet/accuracy_2_of_4‚0.7983990907669067 +jet/accuracy_3_of_4‚0.08900831639766693 )loss/val_total_loss‚9.008392249884336 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984030723571777 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6685553789138794 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9843522906303406 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32224100828170776 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083432674408 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4809873700141907 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3841158151626587 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4487251937389374 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6456294655799866 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505‡Â9ȉë ‚(:þ ÐùöÒäíÞ disk.rbd0.out‚0& disk.nvme1c1n1.out‚10808.166015625$ disk.nvme1n1.out‚10808.166015625% disk./.usageGB‚1173.2971954345703' disk.nvme0c0n1.out‚17674.7666015625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125' proc.memory.availableMB‚ 740.5546875) proc.memory.percent‚17.70329475402832 cpu‚25.674337863718577 proc.cpu.threads‚17 network.recv‚801715) disk./.usagePercent‚76.42385249533108 disk.md0.out‚ 80.82421875 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚23481.36328125! disk.dm-1.out‚4828.0595703125 disk.rbd1.out‚0$ memory_percent‚95.48001289367676 disk.dm-1.in‚0% disk.nvme0n1.out‚17674.7666015625 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2900.5078125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 network.sent‚ 5.103984e+06 disk.md0.in‚0œ%:÷žì ‚(:“ ÐùöÒ¦²‘Þ gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚260.305' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.76833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êL§í‰í ‚(:þ ßùöÒãá€ß disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.98131084442139( proc.memory.availableMB‚ 494.58203125 proc.cpu.threads‚17 network.sent‚ 5.161696e+06 network.recv‚805516 disk.md0.out‚ 80.97265625 disk.dm-0.in‚ 8.64453125! disk.dm-1.out‚4912.1376953125 cpu‚25.674935301291775( disk./.usagePercent‚76.4083514441071 disk.md0.in‚0 disk.nvme0c0n1.in‚7.1875' disk.nvme0c0n1.out‚17675.4658203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0% disk./.usageGB‚1173.0592155456543 disk.dm-1.in‚0% disk.nvme0n1.out‚17675.4658203125& disk.nvme1c1n1.out‚10892.474609375 disk.rbd1.in‚0$ disk.nvme1n1.out‚10892.474609375" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 disk.dm-0.out‚23481.96484375 disk.nvme0n1.in‚7.1875¾?ùžî ‚(:“ ßùöÒÎÓµÞ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç˜Æï ‚(:’ îùöÒ¡ê–Þ gpu.0.gpu‚77.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚242.15' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.71666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“äU…ð ‚(:ú îùöÒˆ…ß disk.md0.out‚ 81.4765625 disk.dm-0.in‚ 8.64453125! disk.dm-0.out‚23486.576171875! disk.dm-1.out‚5073.7216796875 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚17680.443359375 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚17680.443359375" proc.memory.rssMB‚ 2900.5078125% disk./.usageGB‚1173.0592460632324 disk.rbd0.out‚0% disk.nvme1n1.out‚11054.7763671875% proc.memory.availableMB‚ 494.46875) proc.memory.percent‚17.70329475402832 network.recv‚808836( disk./.usagePercent‚76.4083534318992 disk.rbd0.in‚0' disk.nvme1c1n1.out‚11054.7763671875$ memory_percent‚96.98200225830078 cpu‚25.673681521514062 proc.cpu.threads‚17 network.sent‚ 5.16466e+06 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125ß:p2ñ - _wandbruntime‚3077  _runtime‚3077©5 €Éò à _step‚264 ' lr-AdamW/pg1‚0.00016962088833962706 ' lr-AdamW/pg2‚0.00016962088833962706 trainer/global_step‚3249 % _timestamp‚1.7845281136558287e+09 _runtime‚3077.923963106£oO©Îó È ' lr-AdamW/pg1‚0.00016962088833962706 ' lr-AdamW/pg2‚0.00016962088833962706 trainer/global_step‚3249 % _timestamp‚1.7845281136558287e+09 _runtime‚3077.923963106 _step‚264ˆ›F‘¯2ô - _wandbruntime‚3077  _runtime‚3077.‰`â<õ ßO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501242090445824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚498324842518124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58465188 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5674044 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚497466458020352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111941932691456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚494555280484368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32161714 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚497466475059712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108144737031680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13467915 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111942056572928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚498325200118758 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32151571 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚kÜ3Ï© 3775615386112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845281136566086e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3775273961984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58475333 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32161714 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26313619 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3769579073116 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26313617 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32151571 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3797195659776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3077.924760377 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13467634 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501241731982336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108144755643904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚3249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚501242090445824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _step‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚501241731982336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3769919634390 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3775615386112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚497466475059712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3775273961984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7793871 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3797300929024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501242090445824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚498324842518124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58465188 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5674044 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚497466458020352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111941932691456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚494555280484368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32161714 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚497466475059712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108144737031680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13467915 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111942056572928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚498325200118758 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32151571 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3775615386112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845281136566086e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3775273961984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58475333 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32161714 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26313619 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3769579073116 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26313617 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32151571 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3797195659776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3077.924760377 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13467634 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501241731982336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108144755643904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚3249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚501242090445824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _step‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚501241731982336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3769919634390 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3775615386112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚497466475059712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3775273961984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7793871 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3797300929024 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚501391475054080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111976228676608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26321410 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32161227 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5675778 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845281142881947e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498473381573848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26321412 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚501391475054080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚497614732522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3798311374848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58482637 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26321412 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108177917301760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3770687461368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494702711151840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13471644 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3798418734080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚497614749561856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5675775 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498473739174594 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111976354648064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32171384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3776725492224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494702694112480 trainer/global_step‚3249 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚501391116583424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚3078.556472343 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58492796 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58492796 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0”òT)ù îM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26321410 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32161227 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5675778 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845281142881947e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498473381573848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26321412 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚501391475054080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚497614732522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3798311374848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58482637 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26321412 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108177917301760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3770687461368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494702711151840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13471644 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3798418734080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚497614749561856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5675775 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498473739174594 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111976354648064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32171384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3776725492224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494702694112480 trainer/global_step‚3249 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚501391116583424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚3078.556472343 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58492796 ElóËor.on_train_batch_end/inactive_split.small_pool.freed‚7795869 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32161227 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58492796 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚501391475054080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111976228676608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344Šݨdw2ú - _wandbruntime‚3078  _runtime‚3078rÙ˜³ û ƒ 0loss/FRt2/assignment_loss‚1.9520395427068395 2loss/SRqqt2/assignment_loss‚0.5730325203474399 /loss/FRt1/assignment_loss‚0.833202016961043 /loss/FRt1/detection_loss‚1.4534861330865367 /loss/FBt2/detection_loss‚0.7959103275711338 _runtime‚3078.557228754 _step‚267 2loss/SRqqt1/assignment_loss‚0.5350790010308679 /loss/FRt2/detection_loss‚0.5295963002775438 1loss/SRqqt2/detection_loss‚1.0442031479633673 /loss/FBt1/detection_loss‚0.5783966780481703 epoch‚8 trainer/global_step‚3249 2loss/FBt1/assignment_loss‚0.017665392218992722 3loss/FBt2/assignment_loss‚0.0007916469063386463 1loss/SRqqt1/detection_loss‚0.6119854521705863 %loss/total_loss‚8.925388159288861 % _timestamp‚1.7845281142893443e+09ìáñBü ˆ /loss/FRt2/detection_loss‚0.5295963002775438 1loss/SRqqt2/detection_loss‚1.0442031479633673 /loss/FBt1/detection_loss‚0.5783966780481703 epoch‚8 trainer/global_step‚3249 2loss/FBt1/assignment_loss‚0.017665392218992722 3loss/FBt2/assignment_loss‚0.0007916469063386463 1loss/SRqqt1/detection_loss‚0.6119854521705863 %loss/total_loss‚8.925388159288861 % _timestamp‚1.7845281142893443e+09 0loss/FRt2/assignment_loss‚1.9520395427068395 2loss/SRqqt2/assignment_loss‚0.5730325203474399 /loss/FRt1/assignment_loss‚0.833202016961043 /loss/FRt1/detection_loss‚1.4534861330865367 /loss/FBt2/detection_loss‚0.7959103275711338 _runtime‚3078.557228754 _step‚267 2loss/SRqqt1/assignment_loss‚0.5350790010308679‹IèçŠý ‚(:ÿ ýùöÒñ®ðÝ network.recv‚812020$ disk./.usageGB‚1173.059341430664 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚11055.4560546875 network.sent‚ 5.170063e+06) disk./.usagePercent‚76.40835964374949! disk.dm-1.out‚5073.8349609375& disk.nvme0c0n1.out‚17681.466796875 disk.dm-0.in‚ 8.64453125! disk.dm-0.out‚23487.166015625( proc.memory.availableMB‚ 494.37109375) proc.memory.percent‚17.70329475402832 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚17681.466796875 disk.rbd0.out‚0' disk.nvme1c1n1.out‚11055.4560546875 disk.rbd1.out‚0 disk.md0.in‚0" proc.memory.rssMB‚ 2900.5078125 disk.md0.out‚ 81.94921875$ memory_percent‚96.98259830474854 cpu‚25.672460643184074 proc.cpu.threads‚17¥±[4žþ ‚(:“ ýùöÒˆÿ÷Þ gpu.0.gpu‚45.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.841' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.61366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fƒ« zÿ ‚(:ï ŒúöÒ®ÊšÞ disk.nvme1n1.out‚ 11056.6875 disk.md0.in‚0 disk.nvme0c0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125# memory_percent‚96.9827651977539 disk.rbd1.in‚0 disk.md0.out‚ 82.3515625& disk.nvme0c0n1.out‚17693.751953125 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚17693.751953125 disk.rbd0.out‚0) proc.memory.percent‚17.70329475402832 network.sent‚ 5.23393e+06" disk.dm-0.out‚23499.6435546875! disk.dm-1.out‚5074.0166015625! disk.nvme1c1n1.out‚ 11056.6875% proc.memory.availableMB‚ 494.34375" proc.memory.rssMB‚ 2900.5078125 network.recv‚816129% disk./.usageGB‚1173.0592422485352 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.671305777954302 proc.cpu.threads‚17) disk./.usagePercent‚76.40835318342518 disk.rbd1.out‚0dÚt ž€‚(:“ ŒúöÒ¢ŒåÞ gpu.0.gpu‚53.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.304' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.10133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø/BR‚(:ô ›úöÒˇïÝ$ disk.nvme0n1.out‚17694.646484375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 network.sent‚ 5.236894e+06 disk.md0.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0# disk.nvme1c1n1.out‚ 11057.078125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚23500.2763671875 disk.dm-1.in‚0& disk.nvme0c0n1.out‚17694.646484375! disk.nvme1n1.out‚ 11057.078125( proc.memory.availableMB‚ 494.17578125) disk./.usagePercent‚76.40838995757892$ memory_percent‚96.98379039764404" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 cpu‚25.670173516172923 proc.cpu.threads‚17 network.recv‚819806% disk./.usageGB‚1173.0598068237305 disk.md0.out‚82.625! disk.dm-1.out‚5074.0595703125 disk.nvme0c0n1.in‚7.1875nÉá<“‚‚(:ˆ ›úöÒîñÉÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.585' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òòª32ƒ- _wandbruntime‚3122  _runtime‚3122.U;Ç„Á & lr-AdamW/pg1‚0.0001693024224892969 & lr-AdamW/pg2‚0.0001693024224892969 trainer/global_step‚3299 % _timestamp‚1.7845281586607904e+09 _runtime‚3122.928944659 _step‚268==QÌ…Æ % _timestamp‚1.7845281586607904e+09 _runtime‚3122.928944659 _step‚268 & lr-AdamW/pg1‚0.0001693024224892969 & lr-AdamW/pg2‚0.0001693024224892969 trainer/global_step‚3299Œnkì2†- _wandbruntime‚3122  _runtime‚3122 „8øä'‡ÞO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508711384654848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26703269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508711026191360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3825402847798 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚3299 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59339490 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚504880200166912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508711026191360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109803769147904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7909648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3853069738496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501926813857968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3825062286524 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26703269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3831184487936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501926796818608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13670657 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26703267 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32636223 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88492032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7909913 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚3122.930115981 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845281586618986e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3831184487936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113656820274176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26703269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508711026191360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3825402847798 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚3299 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59339490 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚504880200166912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508711026191360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109803769147904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7909648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3853069738496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501926813857968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3825062286524 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26703269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3831184487936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501926796818608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13670657 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26703267 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32636223 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88492032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7909913 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚3122.930115981 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845281586618986e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3831184487936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113656820274176[ÌìE2‰- _wandbruntime‚3123  _runtime‚3123ëñâ& ŠéM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3832294594048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7912196 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113691109773312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826170674776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3831953162752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3854178967552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59367098 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13674389 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344–kÝÉ L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508860769263104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845281592918084e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32656036 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508860410792448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚505900398160856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32645879 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32645879 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3826511236162 _step‚270 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚3299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505028457629696× )Öô&‹îM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚3299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505028457629696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3832294594048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7912196 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113691109773312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826170674776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3831953162752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3854178967552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59367098 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13674389 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508860769263104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845281592918084e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32656036 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508860410792448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚505900398160856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32645879 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32645879 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3826511236162 _step‚270ŽÇUGš2Œ- _wandbruntime‚3123  _runtime‚3123{Ç‚ /loss/FRt2/detection_loss‚0.5397800397242405 1loss/SRqqt1/detection_loss‚0.6612914186382103 0loss/SRqqt2/detection_loss‚0.825834041929832 0loss/FRt2/assignment_loss‚1.8590267466828925 2loss/SRqqt2/assignment_loss‚0.5734288696312543 /loss/FBt1/detection_loss‚0.4476167502175288 %loss/total_loss‚8.999478221580421 0loss/FRt1/assignment_loss‚0.9100982207014705 2loss/SRqqt1/assignment_loss‚0.5653500722935438 /loss/FRt1/detection_loss‚1.3961864264477326 /loss/FBt2/detection_loss‚1.1179753262549639 _step‚271 3loss/FBt2/assignment_loss‚0.0014764833270710653 epoch‚8 trainer/global_step‚3299 % _timestamp‚1.7845281592929683e+09 _runtime‚3123.560840162 1loss/FBt1/assignment_loss‚0.10141382573168115~$) Ž‡ %loss/total_loss‚8.999478221580421 0loss/FRt1/assignment_loss‚0.9100982207014705 2loss/SRqqt1/assignment_loss‚0.5653500722935438 /loss/FRt1/detection_loss‚1.3961864264477326 /loss/FBt2/detection_loss‚1.1179753262549639 _step‚271 3loss/FBt2/assignment_loss‚0.0014764833270710653 epoch‚8 trainer/global_step‚3299 % _timestamp‚1.7845281592929683e+09 _runtime‚3123.560840162 1loss/FBt1/assignment_loss‚0.10141382573168115 /loss/FRt2/detection_loss‚0.5397800397242405 1loss/SRqqt1/detection_loss‚0.6612914186382103 0loss/SRqqt2/detection_loss‚0.825834041929832 0loss/FRt2/assignment_loss‚1.8590267466828925 2loss/SRqqt2/assignment_loss‚0.5734288696312543 /loss/FBt1/detection_loss‚0.4476167502175288¨!Ž‚(:ƒ ªúöÒµ“«Þ disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0( proc.memory.availableMB‚ 494.08984375" proc.memory.rssMB‚ 2900.5078125 proc.cpu.threads‚17& disk.nvme0n1.out‚17697.11865234375) disk./.usagePercent‚76.39294083742342# disk.dm-0.out‚23502.98876953125& disk.nvme1n1.out‚11057.86767578125) proc.memory.percent‚17.70329475402832 cpu‚25.66903809915252 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0( disk.nvme1c1n1.out‚11057.86767578125 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.98431491851807( disk.nvme0c0n1.out‚17697.11865234375 network.sent‚ 5.255709e+06 network.recv‚822987 disk.md0.in‚0 disk.md0.out‚82.8125 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚5074.19091796875 disk.rbd0.in‚0$ disk./.usageGB‚1172.822624206543–ÍUž‚(:“ ªúöÒ ˆçÞ gpu.0.gpu‚39.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.297' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.43233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zƒx*“‘‚(:ˆ ¹úöÒÒÔ“Þ disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚17698.00439453125 disk.rbd0.out‚0( disk.nvme1c1n1.out‚11058.24951171875) disk./.usagePercent‚76.37746885765881 disk.md0.out‚ 83.05859375 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚23503.67626953125 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0& disk.nvme1n1.out‚11058.24951171875$ memory_percent‚96.98431491851807 network.sent‚ 5.305961e+06" disk.dm-1.out‚5074.22607421875 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 cpu‚25.668082524906758$ disk./.usageGB‚1172.585090637207( disk.nvme0c0n1.out‚17698.00439453125( proc.memory.availableMB‚ 494.08984375" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 proc.cpu.threads‚17 network.recv‚827097 disk.md0.in‚0 disk.dm-1.in‚0s›¦“’‚(:ˆ ¹úöÒÖ÷ÌÞ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.675' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x]õ3“‚(:’ ÈúöÒ鳑Þ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.89' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.29666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ÅöªŽ”‚(:ƒ ÈúöÒÑÅäÞ$ memory_percent‚96.98433876037598" proc.memory.rssMB‚ 2900.5078125 disk.dm-0.in‚ 8.64453125 network.recv‚830330 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0 cpu‚25.667043608010225 proc.cpu.threads‚17 disk.rbd1.out‚0# disk.dm-0.out‚23509.56591796875 disk.nvme0n1.in‚7.1875' proc.memory.availableMB‚ 494.0859375) proc.memory.percent‚17.70329475402832 network.sent‚ 5.308984e+06% disk./.usageGB‚1172.5853538513184 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚11058.98486328125 disk.md0.in‚0 disk.md0.out‚83.375" disk.dm-1.out‚5074.34716796875( disk.nvme0c0n1.out‚17703.96142578125) disk./.usagePercent‚76.37748600236563( disk.nvme1c1n1.out‚11058.98486328125 disk.rbd1.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚17703.96142578125 disk.rbd0.in‚0H%R2•- _wandbruntime‚3167  _runtime‚31672“ÑÉ–Ã ' lr-AdamW/pg2‚0.00016897882402618365 trainer/global_step‚3349 % _timestamp‚1.7845282028888783e+09 _runtime‚3167.157089429 _step‚272 ' lr-AdamW/pg1‚0.00016897882402618365.ÃVãÎ—È ' lr-AdamW/pg1‚0.00016897882402618365 ' lr-AdamW/pg2‚0.00016897882402618365 trainer/global_step‚3349 % _timestamp‚1.7845282028888783e+09 _runtime‚3167.157089429 _step‚272ÊÑm2˜- _wandbruntime‚3167  _runtime‚3167h“Y#™ßO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚111462764039680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512293925274112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516180320400384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33131018 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33120875 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024929 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚513179233292774 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚516180678863872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33131018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3886412165632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60213792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516180320400384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27092919 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27092917 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3880886061206 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512293925274112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33120875 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111462782651904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5847441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8024655 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3886753589760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512293908234752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13872373 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚513178875692140 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60223937 _runtime‚3167.158177201 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13872096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115371797484544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512293908234752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516180678863872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3909014832640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115371675700224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_p kieÑool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516180320400384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27092919 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27092917 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3880886061206 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512293925274112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33120875 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111462782651904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5847441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8024655 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3886753589760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512293908234752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13872373 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚513178875692140 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60223937 _runtime‚3167.158177201 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13872096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115371797484544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512293908234752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516180678863872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3909014832640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115371675700224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33120875 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024929 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚513179233292774 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚516180678863872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33131018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3886412165632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0‘¾¦‡'2›- _wandbruntime‚3167  _runtime‚3167¹´î&œèM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13876120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512442199776256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5849175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60241400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509445777899040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3887522264576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8027227 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3887522264576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3887863695872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60241400 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3881653888184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516329705001472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 $ _timestamp‚1.784528203514592e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111495962921984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33130531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115405967163904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512442182736896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8026945 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115406091038208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3881994449570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚3349 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3887863695872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13876405 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚513327772348610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27100712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60231241 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27100710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512442199776256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509445760859680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512442182736896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 h”‰ó&íM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27100712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60231241 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27100710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512442199776256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509445760859680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512442182736896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13876120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512442199776256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5849175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60241400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509445777899040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3887522264576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8027227 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3887522264576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3887863695872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60241400 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3881653888184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516329705001472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 $ _timestamp‚1.784528203514592e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111495962921984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33130531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115405967163904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512442182736896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8026945 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115406091038208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3881994449570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚3349 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3887863695872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13876405 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚513327772348610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285’"?,ø2ž- _wandbruntime‚3167  _runtime‚3167-KègŸ 0loss/FRt1/assignment_loss‚0.8529752679899627 /loss/FRt1/detection_loss‚1.4003881859163954 1loss/SRqqt2/detection_loss‚0.9792187652783468 % _timestamp‚1.7845282035157127e+09 _step‚275 0loss/FRt2/assignment_loss‚2.0056109963579383 2loss/SRqqt1/assignment_loss‚0.5193374835598469 /loss/FBt2/detection_loss‚1.2398136001494195 %loss/total_loss‚9.319061286850463 epoch‚8 2loss/SRqqt2/assignment_loss‚0.5865187021527375 1loss/FBt1/assignment_loss‚0.10106247228704252 /loss/FRt2/detection_loss‚0.5813502600105378 0loss/FBt1/detection_loss‚0.40729779442825803 trainer/global_step‚3349 _runtime‚ 3167.7834997 2loss/FBt2/assignment_loss‚0.009940606921426716 1loss/SRqqt1/detection_loss‚0.6355471517985509ʃßpÁ † 0loss/FRt1/assignment_loss‚0.8529752679899627 /loss/FRt1/detection_loss‚1.4003881859163954 1loss/SRqqt2/detection_loss‚0.9792187652783468 % _timestamp‚1.7845282035157127e+09 _step‚275 0loss/FRt2/assignment_loss‚2.0056109963579383 2loss/SRqqt1/assignment_loss‚0.5193374835598469 /loss/FBt2/detection_loss‚1.2398136001494195 %loss/total_loss‚9.319061286850463 epoch‚8 2loss/SRqqt2/assignment_loss‚0.I^…zK5865187021527375 1loss/FBt1/assignment_loss‚0.10106247228704252 /loss/FRt2/detection_loss‚0.5813502600105378 0loss/FBt1/detection_loss‚0.40729779442825803 trainer/global_step‚3349 _runtime‚ 3167.7834997 2loss/FBt2/assignment_loss‚0.009940606921426716 1loss/SRqqt1/detection_loss‚0.6355471517985509“Š¬*Ž¡‚(:ƒ ×úöÒž×ÜÞ disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2900.5078125" disk.dm-1.out‚5074.44482421875 disk.rbd0.out‚0 disk.rbd1.out‚0) proc.memory.percent‚17.70329475402832 cpu‚25.666026102262652 network.recv‚833881 disk.md0.in‚0# disk.dm-0.out‚23542.84716796875 disk.dm-1.in‚0 disk.rbd1.in‚0& disk.nvme1n1.out‚11091.48876953125# proc.memory.availableMB‚496.125 proc.cpu.threads‚17% disk./.usageGB‚1172.8879013061523 disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚17705.43017578125 disk.rbd0.in‚0( disk.nvme1c1n1.out‚11091.48876953125$ memory_percent‚96.97189331054688 network.sent‚ 5.326727e+06) disk./.usagePercent‚76.39719272471274 disk.md0.out‚ 83.6484375( disk.nvme0c0n1.out‚17705.430175781252 ž¢‚(:“ ×úöÒšý ß gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.745' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.58166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ÙjoŒ£‚(: æúöÒÈÓþÝ# proc.memory.availableMB‚495.875) proc.memory.percent‚17.70329475402832 proc.cpu.threads‚17 disk.nvme1c1n1.in‚ 1.45703125& disk.nvme1n1.out‚11115.88623046875 disk.md0.out‚ 84.328125# disk.dm-0.out‚23566.83935546875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚17706.63525390625 disk.nvme0n1.in‚7.1875) disk./.usagePercent‚76.38171801173401% disk./.usageGB‚1172.6503257751465 disk.rbd0.in‚0( disk.nvme1c1n1.out‚11115.88623046875 disk.nvme1n1.in‚ 1.45703125" disk.dm-1.out‚5074.62158203125 disk.nvme0c0n1.in‚7.1875" proc.memory.rssMB‚ 2900.5078125 network.sent‚ 5.378197e+06 network.recv‚837863 disk.rbd1.out‚0 disk.md0.in‚0& disk.nvme0n1.out‚17706.63525390625 cpu‚25.66502935313371 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 8.64453125$ memory_percent‚96.97341918945312@-@ž¤‚(:“ æúöÒðì½Þ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚281.363' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.78766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù–ê Ž¥‚(:ƒ õúöÒªŽÞ disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚17883.89794921875 disk.nvme0n1.in‚7.1875# memory_percent‚96.9719409942627 proc.cpu.threads‚17 network.sent‚ 5.382142e+06$ disk./.usageGB‚1172.586769104004 disk.md0.out‚ 84.54296875# disk.dm-0.out‚23772.17919921875" disk.dm-1.out‚5074.67236328125' proc.memory.availableMB‚ 496.1171875 cpu‚25.66402760259361' disk./.usagePercent‚76.377578186224 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.in‚0& disk.nvme0n1.out‚17883.89794921875) proc.memory.percent‚17.70329475402832( disk.nvme1c1n1.out‚11144.49267578125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚11144.49267578125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2900.5078125 network.recv‚840627 disk.rbd1.in‚04C|ž¦‚(:“ õúöÒœ˜ËÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚122.641' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.88033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$úÔ2§- _wandbruntime‚3211  _runtime‚32112ó¬iȨ ' lr-AdamW/pg1‚0.00016865011373441195 ' lr-AdamW/pg2‚0.00016865011373441195 trainer/global_step‚3399 $ _timestamp‚1.784528246788754e+09 _runtime‚3211.057526551 _step‚276j}è¢Í©Ç _runtime‚3211.057526551 _step‚276 ' lr-AdamW/pg1‚0.00016865011373441195 ' lr-AdamW/pg2‚0.00016865011373441195 trainer/global_step‚3399 $ _timestamp‚1.784528246788754e+09” жª2ª- _wandbruntime‚3211  _runtime‚3211¬[kä'«ÞO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚523649614609408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113121796155904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27482567 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33615670 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚516669880605168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚519707633341952 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3936369274614 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27482569 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33615670 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8140271 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519707650381312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3942322691584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33605527 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113121777543680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 $ _timestamp‚1.784528246789521e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8139971 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520605892279148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚3399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14074415 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5934141 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚523649973072896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3964748249600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3936028713340 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚516669863565808 _step‚277 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61088094 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚519707650381312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3941981267456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520606249879782 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27482569 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3942322691584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117086525793280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33605527 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117086660160512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚519707633341952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61098239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚3211.057695082 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534à![é'¬ãO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3936369274614 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27482569 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33615670 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8140271 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519707650381312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3942322691584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33605527 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113121777543680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 $ _timestamp‚1.784528246789521e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8139971 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520605892279148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚3399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14074415 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5934141 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚523649973072896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3964748249600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3936028713340 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚516669863565808 _step‚277 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61088094 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚519707650381312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3941981267456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520606249879782 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27482569 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3942322691584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117086525793280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33605527 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117086660160512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚519707633341952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61098239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚3211.057695082 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚523649614609408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113121796155904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27482567 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33615670 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚516669880605168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚519707633341952 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145•öÜ°2­- _wandbruntime‚3211  _runtime‚3211ª¦5Ò®çM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14078410 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3965871999488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33615183 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8142244 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚523799357681152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3943432797696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5935878 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚519855907844096 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 3211.68652658 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27490362 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚520754431334872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845282474182591e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3943091366400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚519855924883456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61105543 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113154957813760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33625340 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3943091366400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61115702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27490360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚278 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33625340 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14078119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚516817311272640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ýI ò&¯ìM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚516817311272640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14078410 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3965871999488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33615183 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8142244 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚523799357681152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3943432797696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5935878 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚519855907844096 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 3211.68652658 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27490362 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚520754431334872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845282474182591e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3943091366400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚519855924883456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61105543 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113154957813760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33625340 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3943091366400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61115702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27490360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚278 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33625340 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14078119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚3449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚531118908818432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8255694 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524041396939408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61962396 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3997891793408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3997550369280 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531118908818432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚531119267281920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118801522484224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61972541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114780809659904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3991852488022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3997891793408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34100322 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27872219 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚527121358449152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114780791047680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚527121358449152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3991511926748 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚527121375488512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚531119267281920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845282915437565e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0!.”5Ù¾äO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34090179 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34100322 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34090179 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚527121375488512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8255411 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14276252 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚3449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚531118908818432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8255694 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524041396939408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61962396 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3997891793408 ZHDeviceStatsMonitor.o¹Ë[ n_train_batch_start/allocated_bytes.small_pool.freed‚ 3997550369280 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531118908818432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚531119267281920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118801522484224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61972541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114780809659904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3991852488022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3997891793408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34100322 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27872219 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚527121358449152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114780791047680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚527121358449152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3991511926748 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚527121375488512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚531119267281920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845282915437565e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6020841 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6020844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118801396505600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528033266466790 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚3255.811835594 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3997550369280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528032908866156 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4020712824320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4020605457920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0™cgä2¿- _wandbruntime‚3256  _runtime‚3256¨NE‹ï&ÀéM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3998660468224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3999001899520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚527269632951296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 % _timestamp‚1.7845282921708229e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚282 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6022575 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114813989929984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14280256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3999001899520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8257981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531268651890176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27880010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34099835 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114813971317760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531268293419520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14280559 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61990004 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528181447921880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚527269649990656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3992960876386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272†H¯Wô&ÁîM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3992960876386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3998660468224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3999001899520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚527269632951296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 % _timestamp‚1.7845282921708229e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚282 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6022575 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114813989929984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14280256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3999001899520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8257981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531268651890176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27880010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34099835 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114813971317760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531268293419520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14280559 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61990004 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528181447921880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚527269649990656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0šŠ†ç2Â- _wandbruntime‚3256  _runtime‚3256àCÈ 1loss/SRqqt2/detection_loss‚1.0292553968932847 % _timestamp‚1.7845282921719134e+09 _step‚283 0loss/FRt1/assignment_loss‚0.8083753641804137 /loss/FRt2/detection_loss‚0.5052262776980883 %loss/total_loss‚8.935141400337844 0loss/FBt1/detection_loss‚0.44748011960892503 /loss/FBt2/detection_loss‚1.0255946227379384 epoch‚8 trainer/global_step‚3449 1loss/SRqqt2/assignment_loss‚0.647275180799864 1loss/FBt1/assignment_loss‚0.11053561784822567 2loss/FBt2/assignment_loss‚0.023640798321733176 1loss/SRqqt1/detection_loss‚0.6051868043373926 _runtime‚3256.440191714 0loss/FRt2/assignment_loss‚1.9040784167520928 2loss/SRqqt1/assignment_loss‚0.5503411652117774 /loss/FRt1/detection_loss‚1.2781516359481093Ûc³þ ć 1loss/SRqqt2/assignment_loss‚0.647275180799864 1loss/FBt1/assignment_loss‚0.11053561784822567 2loss/FBt2/assignment_loss‚0.023640798321733176 1loss/SRqqt1/detection_loss‚0.6051868043373926 _runtime‚3256.440191714 0loss/FRt2/assignment_loss‚1.9040784167520928 2loss/SRqqt1/assignment_loss‚0.5503411652117774 /loss/FRt1/detection_loss‚1.2781516359481093 1loss/SRqqt2/detection_loss‚1.0292553968932847 % _timestamp‚1.7845282921719134e+09 _step‚283 0loss/FRt1/assignment_loss‚0.8083753641804137 /loss/FRt2/detection_loss‚0.5052262776980883 %loss/total_loss‚8.935141400337844 0loss/FBt1/detection_loss‚0.44748011960892503 /loss/FBt2/detection_loss‚1.0255946227379384 epoch‚8 trainer/global_step‚3449›u5›ÎŽÅ‚(:ƒ ±ûöÒñ„èÝ) proc.memory.percent‚17.70329475402832 cpu‚25.660961734764232 network.recv‚854733) disk./.usagePercent‚76.37752898836968% disk.nvme0n1.out‚18312.3388671875 disk.dm-0.in‚ 8.64453125 disk.dm-1.out‚5075.232421875' disk.nvme0c0n1.out‚18312.3388671875( proc.memory.availableMB‚ 495.91796875 proc.cpu.threads‚17 disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚11249.12451171875 disk.nvme0c0n1.in‚7.1875$ memory_percent‚96.97315692901611 network.sent‚ 5.457235e+06 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚11249.12451171875 disk.md0.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1172.5860137939453 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 disk.md0.out‚ 86.109375# disk.dm-0.out‚24301.32958984375" proc.memory.rssMB‚ 2900.5078125­]¬°Æ‚(:’ ±ûöÒîÏÐÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.344' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.114666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>F¿aÇ‚(:„ ÀûöÒ–Š…Þ' disk.nvme0c0n1.out‚18432.2802734375 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0) proc.memory.percent‚17.70329475402832 network.recv‚859017 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚11340.31396484375 disk.md0.in‚0% disk.nvme0n1.out‚18432.2802734375 proc.cpu.threads‚17) disk./.usagePercent‚76.39303177891172" disk.dm-1.out‚5075.39013671875 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0( proc.memory.availableMB‚ 495.41796875 network.sent‚ 5.522087e+06% disk./.usageGB‚1172.8240203857422( disk.nvme1c1n1.out‚11340.31396484375$ memory_percent‚96.97620868682861" proc.memory.rssMB‚ 2900.5078125 cpu‚25.660075236253633 disk.md0.out‚ 86.484375 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚24511.4970703125 disk.dm-1.in‚0¥³ RžÈ‚(:“ ÀûöÒÜîÉÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚286.906' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.63533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¿f{2É- _wandbruntime‚3299  _runtime‚3299Т_ÉÉÊà ' lr-AdamW/pg1‚0.00016797744244165147 ' lr-AdamW/pg2‚0.00016797744244165147 trainer/global_step‚3499 % _timestamp‚1.7845283356671124e+09 _runtime‚3299.935274662 _step‚284ΨUnÎËÈ trainer/global_step‚3499 % _timestamp‚1.7845283356671124e+09 _runtime‚3299.935274662 _step‚284 ' lr-AdamW/pg1‚0.00016797744244165147 ' lr-AdamW/pg2‚0.00016797744244165147œ—&ͤ2Ì- _wandbruntime‚3299  _runtime‚3299³=>õÐÍßO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28261869 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚538588561490944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚3499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34584974 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚538588561490944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538588203027456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytX#ðq"es.large_pool.freed‚534535083556352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8370786 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4076471268864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14478043 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62846843 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14478330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4053460895232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28261867 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116439823163904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3299.939611814 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6107541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538588203027456 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28261869 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚535459925453164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4046995140156 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62836698 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4053119471104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8370502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120516395507712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28261867 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4053460895232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4076572343808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534535100595712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116439804551680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4047335701430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531412947352368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚534535100595712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34574831 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6107544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538588203027456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534535083556352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8370786 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4076471268864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14478043 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62846843 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14478330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4053460895232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28261867 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116439823163904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3299.939611814 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6107541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538588203027456 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28261869 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚535459925453164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4046995140156 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62836698 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4053119471104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8370502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120516395507712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28261867 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4053460895232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4076572343808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534535100595712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116439804551680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4047335701430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531412947352368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚534535100595712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34574831 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6107544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 B‰Ï‚(:þ ÏûöÒøÎÞ" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 proc.cpu.threads‚17 network.sent‚ 5.525009e+06 disk.md0.in‚0$ disk.nvme0c0n1.out‚ 18446.6484375 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚96.97768688201904 cpu‚25.659048612906346 network.recv‚862813 disk.md0.out‚ 86.8203125! disk.dm-0.out‚24528.666015625" disk.dm-1.out‚5075.40966796875" disk.nvme0n1.out‚ 18446.6484375 disk.rbd0.in‚0( disk.nvme1c1n1.out‚11343.85302734375 disk.nvme1n1.in‚ 1.45703125) disk./.usagePercent‚76.39308072829203% disk./.usageGB‚1172.8247718811035 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0& disk.nvme1n1.out‚11343.85302734375 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0( proc.memory.availableMB‚ 495.17578125Â}é4žÐ‚(:“ ÏûöÒéÙÉÞ gpu.0.gpu‚92.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.269' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.75633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nd^¾2Ñ- _wandbruntime‚3300  _runtime‚3300ê¬_ýî&ÒèM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚534683358058496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4054571001344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚535608822109634 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28269660 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62864306 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62854147 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62854147 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14482063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28269662 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116473003433984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531560360980480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚3300.563338646 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4054229570048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚538737587628544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚538737946099200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28269660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4048444089794 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4077591725568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28269662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534683375097856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4048103528408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4054571001344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14482337 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 % _timestamp‚1.7845283362950995e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62864306 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62854147 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62854147 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14482063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28269662 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116473003433984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531560360980480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚3300.563338646 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4054229570048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚538737587628544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚538737946099200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28269660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4048444089794 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4077591725568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28269662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534683375097856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4048103528408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4054571001344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14482337 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386žòýõa2Ô- _wandbruntime‚3300  _runtime‚3300˼â—Õ _step‚287 trainer/global_step‚3499 _runtime‚3300.563871037 0loss/FRt1/assignment_loss‚0.7773170426251397 /loss/FRt2/detection_loss‚0.5375492644016823 0loss/SRqqt1/detection_loss‚0.638634003498305 /loss/FBt1/detection_loss‚0.5865728328727544 /loss/FBt2/detection_loss‚1.0644524387316778 2loss/SRqqt1/assignment_loss‚0.5646364539090157 /loss/FRt1/detection_loss‚1.3496661538522439 epoch‚8 0loss/FRt2/assignment_loss‚1.9497905802219473 2loss/SRqqt2/assignment_loss‚0.6815780256475762 2loss/FBt1/assignment_loss‚0.052325939639042626 2loss/FBt2/assignment_loss‚0.007601635252667771 1loss/SRqqt2/detection_loss‚0.9022167089138573 $loss/total_loss‚9.11234107956591 % _timestamp‚1.7845283362960932e+09}§’Ö Ö† 1loss/SRqqt2/detection_loss‚0.9022167089138573 $loss/total_loss‚9.11234107956591 % _timestamp‚1.7845283362960932e+09 _step‚287 trainer/global_step‚3499 _runtime‚3300.563871037 0loss/FRt1/assignment_loss‚0.7773170426251397 /loss/FRt2/detection_loss‚0.5375492644016823 0loss/SRqqt1/detection_loss‚0.638634003498305 /loss/FBt1/detection_loss‚0.5865728328727544 /loss/FBt2/detection_loss‚1.0644524387316778 2loss/SRqqt1/assignment_loss‚0.5646364539090157 /loss/FRt1/detection_loss‚1.3496661538522439 epoch‚8 0loss/FRt2/assignment_loss‚1.9497905802219473 2loss/SRqqt2/assignment_loss‚0.6815780256475762 2loss/FBt1/assignment_loss‚0.052325939639042626 2loss/FBt2/assignment_loss‚0.007601635252667771Ÿ›Æn‰×‚(:þ ÞûöÒ÷ˬÞ$ memory_percent‚96.97690010070801) disk./.usagePercent‚76.39305463852081% disk./.usageGB‚1172.8243713378906 disk.rbd0.in‚0! disk.dm-1.out‚5075.5869140625" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 cpu‚25.658103437744646 network.sent‚ 5.589581e+06 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125$ disk.nvme1n1.out‚11344.458984375 disk.md0.in‚0' disk.nvme0c0n1.out‚18481.6552734375' proc.memory.availableMB‚ 495.3046875 network.recv‚866849% disk.nvme0n1.out‚18481.6552734375 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.out‚ 87.1015625! disk.dm-0.out‚24563.470703125 proc.cpu.threads‚17 disk.rbd0.out‚0& disk.nvme1c1n1.out‚11344.458984375 disk.rbd1.out‚0 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875‡Ò•ŸØ‚(:” ÞûöÒ£´íÞ gpu.0.gpu‚58.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚128.529' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚42.842999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òУwˆÙ‚(:ý íûöÒÜ’ˆÞ proc.cpu.threads‚17% disk./.usageGB‚1173.0493545532227 disk.nvme0c0n1.in‚7.1875& disk.nvme0c0n1.out‚18509.576171875 disk.nvme0n1.in‚7.1875 disk.rbd1.out‚0( disk./.usagePercent‚76.4077091387867 disk.md0.in‚0! disk.dm-0.out‚24592.677734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚11346.4970703125 disk.rbd1.in‚0 network.sent‚ 5.62875e+06 network.recv‚870897 disk.md0.out‚ 87.42578125 disk.dm-1.in‚0! disk.dm-1.out‚5075.6455078125% disk.nvme1n1.out‚11346.4970703125$ memory_percent‚96.97699546813965' proc.memory.availableMB‚ 495.2890625 disk.dm-0.in‚ 8.64453125$ disk.nvme0n1.out‚18509.576171875 disk.nvme1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 cpu‚25.657175158758214 ä¶h“Ú‚(:ˆ íûöÒâ†ÖÞ gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.118' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è¿må2Û- _wandbruntime‚3344  _runtime‚3344|^Ï"ÉÜà ' lr-AdamW/pg1‚0.00016763352464508287 ' lr-AdamW/pg2‚0.00016763352464508287 trainer/global_step‚3549 % _timestamp‚1.7845283805855176e+09 _runtime‚3344.853644613 _step‚2883|£ÎÝÈ ' lr-AdamW/pg1‚0.00016763352464508287 ' lr-AdamW/pg2‚0.00016763352464508287 trainer/global_step‚3549 % _timestamp‚1.7845283805855176e+09 _runtime‚3344.853644613 _step‚288 t&Æ:2Þ- _wandbruntime‚3344  _runtime‚3344èÀÞå'ßßO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4102818914838 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚541948808663552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28651519 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35069626 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4108688572928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542887299640806 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122231156923904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542886942040172 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109029997056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35069626 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28651519 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35059483 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35059483 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6194241 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538784463686608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14679544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚541948825702912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118098818055680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚289 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546057855699968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546057497236480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚541948808663552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4132338868224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 % _timestamp‚1.7845283805865622e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚538784480725968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6194244 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8485303 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109029997056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚3344.854780825 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63711000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122231280805376G+ûžÌ'àäO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122231280805376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4102818914838 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚541948808663552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28651519 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35069626 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4108688572928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542887299640806 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122231156923904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542886942040172 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109029997056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35069626 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28651519 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35059483 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35059483 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6194241 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538784463686608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14679544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚541948825702912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118098818055680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚289 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546057855699968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546057497236480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚541948808663552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4132338868224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 % _timestamp‚1.7845283805865622e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚538784480725968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6194244 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8485303 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109029997056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚3344.854780825 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63711000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_starH'Æ t/num_device_alloc‚1452¡XV'Ñ‹á‚(:€ üûöÒ‹ìÊÞ disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.42406866772131% disk./.usageGB‚1173.3005142211914 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125$ disk.nvme1n1.out‚11399.583984375 disk.dm-0.in‚ 8.64453125 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 network.recv‚874347 disk.rbd1.out‚0 disk.md0.in‚0' disk.nvme0c0n1.out‚18861.2041015625$ memory_percent‚96.97847366333008& proc.memory.availableMB‚ 495.046875" proc.memory.rssMB‚ 2900.5078125 cpu‚25.656242303435953 network.sent‚ 5.647586e+06 disk.nvme1c1n1.in‚ 1.45703125& disk.nvme1c1n1.out‚11399.583984375# disk.dm-0.out‚24996.09033203125" disk.dm-1.out‚5075.85302734375 disk.nvme0c0n1.in‚7.1875% disk.nvme0n1.out‚18861.2041015625) proc.memory.percent‚17.70329475402832 disk.md0.out‚ 87.9453125 disk.dm-1.in‚0ÏŽ'kžâ‚(:“ üûöÒö«‡ß gpu.0.gpu‚36.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.812' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à»[$2ã- _wandbruntime‚3345  _runtime‚33452UFûî&äèM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109798671872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35079296 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚543035481095896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28659310 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4133564375552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚543035838696642 _step‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118132016937984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14683531 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4103927303202 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚542097100205056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚3549 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8487843 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35079296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122265449050624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35069139 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28659312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6195975 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚538931911393440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63728449 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8487556 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122265581313536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546206881837568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542097083165696 $ _timestamp‚1.784528381216408e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63728449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28659310 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63738608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109798671872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚546207240308224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 _runtime‚3345.484972537 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35069139 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14683821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚542097083165696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63738608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚538931894354080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4110140103168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4133450724864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4110140103168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542097100205056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546207240308224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28659312,iˆxó&åíM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63728449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28659310 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63738608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109798671872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚546207240308224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 _runtime‚3345.484972537 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35069139 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14683821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚542097083165696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63738608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚538931894354080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4110140103168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4133450724864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4110140103168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542097100205056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546207240308224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28659312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109798671872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35079296 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚543035481095896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28659310 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4133564375552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚543035838696642 _step‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118132016937984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14683531 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4103927303202 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚542097100205056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚3549 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8487843 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35079296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122265449050624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35069139 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28659312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6195975 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚538931911393440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63728449 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8487556 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122265581313536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546206881837568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542097083165696 $ _timestamp‚1.784528381216408e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344¢|"ðû2æ- _wandbruntime‚3345  _runtime‚3345E;çÿ trainer/global_step‚3549 _runtime‚3345.485650348 -loss/FBt2/detection_loss‚1.18451078038197 $loss/total_loss‚9.26097210231833 2loss/FBt2/assignment_loss‚0.013582822582080212 /loss/FRt1/detection_loss‚1.3540123327370934 /loss/FBt1/detection_loss‚0.5655877865771948 epoch‚8 $ _timestamp‚1.784528381217835e+09 2loss/SRqqt2/assignment_loss‚0.5058435823063269 2loss/FBt1/assignment_loss‚0.028146266627554126 1loss/SRqqt1/detection_loss‚0.6403535604508308 1loss/SRqqt2/detection_loss‚1.0285839682496851 _step‚291 0loss/FRt1/assignment_loss‚0.8750794261809022 0loss/FRt2/assignment_loss‚1.9580209476792294 2loss/SRqqt1/assignment_loss‚0.5437064340402123 /loss/FRt2/detection_loss‚0.5635441945052504º×Ÿñ è„ 2loss/FBt2/assignment_loss‚0.013582822582080212 /loss/FRt1/detection_loss‚1.3540123327370934 /loss/FBt1/detection_loss‚0.5655877865771948 epoch‚8 $ _timestamp‚1.784528381217835e+09 2loss/SRqqt2/assignment_loss‚0.5058435823063269 2loss/FBt1/assignment_loss‚0.028146266627554126 1loss/SRqqt1/detection_loss‚0.6403535604508308 1loss/SRqqt2/detection_loss‚1.0285839682496851 _step‚291 0loss/FRt1/assignment_loss‚0.8750794261809022 0loss/FRt2/assignment_loss‚1.9580209476792294 2loss/SRqqt1/assignment_loss‚0.5437064340402123 /loss/FRt2/detection_loss‚0.5635441945052504 trainer/global_step‚3549 _runtime‚3345.485650348 -loss/FBt2/detection_loss‚1.18451078038197 $loss/total_loss‚9.26097210231833£®}‰é‚(:þ ‹üöÒ“áÞ" proc.memory.rssMB‚ 2900.5078125) proc.memory.percent‚17.70329475402832 proc.cpu.threads‚17& disk.nvme0c0n1.out‚18904.189453125$ disk.nvme0n1.out‚18904.189453125 disk.md0.out‚ 88.1640625 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.45703125( proc.memory.availableMB‚ 495.01953125 network.recv‚878515" disk.dm-1.out‚5075.93896484375 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0$ memory_percent‚96.97864055633545 cpu‚25.655405237714387 network.sent‚ 5.69816e+06( disk./.usagePercent‚76.4084866139695% disk./.usageGB‚1173.0612907409668 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚25042.05615234375 disk.dm-1.in‚0& disk.nvme1c1n1.out‚11403.134765625 disk.rbd1.in‚0 disk.md0.in‚0 disk.rbd0.out‚0$ disk.nvme1n1.out‚11403.134765625Ô;×Bê‚(:’ ‹üöÒýæÜÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.89' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.29666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œB¢Ù†ë‚(:û šüöÒÕÂùÝ% disk./.usageGB‚1173.0616607666016 disk.dm-0.in‚ 8.64453125! disk.dm-1.out‚5076.0185546875( disk.nvme0c0n1.out‚18943.88525390625 disk.rbd0.out‚0 disk.nvme1n1.out‚ 11403.53125 network.recv‚882537) disk./.usagePercent‚76.40851071594864 disk.md0.out‚ 88.39453125 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚18943.88525390625" disk.nvme1c1n1.out‚ 11403.53125 disk.rbd1.in‚0 network.sent‚ 5.737387e+06 disk.nvme0c0n1.in‚7.1875 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.98071479797363' proc.memory.availableMB‚ 494.6796875) proc.memory.percent‚17.70329475402832 cpu‚25.654428086430794 disk.md0.in‚0# disk.dm-0.out‚25081.54443359375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2900.5078125 proc.cpu.threads‚17YUì‚(:’ šüöÒ¼½ÐÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚129.14' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.04666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 °Q2í- _wandbruntime‚3389  _runtime‚3389<’ùíÉîà _step‚292 ' lr-AdamW/pg1‚0.00016728458142592116 ' lr-AdamW/pg2‚0.00016728458142592116 trainer/global_step‚3599 % _timestamp‚1.7845284254255674e+09 _runtime‚3389.694606397¢N€ÎïÈ ' lr-AdamW/pg1‚0.00016728458142592116 ' lr-AdamW/pg2‚0.00016728458142592116 trainer/global_step‚3599 % _timestamp‚1.7845284254255674e+09 _runtime‚3389.694606397 _step‚292¤è#C·2ð- _wandbruntime‚3389  _runtime‚3389‰¹&Y£ñÝO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4164257674752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29041169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚546155997060208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚293 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35544135 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550314316227814 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549362550810112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6280944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119757831559680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚549362533770752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14881027 trainer/global_step‚3599 $ _timestamp‚1.784528425426262e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29041167 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4157961566972 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29041167 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553526791445504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14881313 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64585302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚549362550810112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8600369 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3389.694688248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29041169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553527149908992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64595447 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123946132250112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚546156014099568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.ͧÊ@allocated‚35554278 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4158302128246 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553527149908992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123946016757248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35544135 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550313958627180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119757850171904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4188282078208 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553526791445504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4188185197568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4164257674752 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6280941 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14881027 trainer/global_step‚3599 $ _timestamp‚1.784528425426262e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29041167 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4157961566972 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29041167 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553526791445504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14881313 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64585302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚549362550810112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8600369 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3389.694688248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29041169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553527149908992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64595447 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123946132250112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚546156014099568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35554278 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4158302128246 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553527149908992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123946016757248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35544135 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550313958627180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119757850171904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4188282078208 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553526791445504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4188185197568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4164257674752 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6280941 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549362550810112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6280944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119757831559680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚549362533770752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¥p®òƒó‚(:ø ©üöÒªÔ”Þ" proc.memory.rssMB‚ 2900.5078125 cpu‚25.65359783986834 network.recv‚886030% disk./.usageGB‚1173.0614166259766 disk.nvme1c1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875 proc.cpu.threads‚17( disk./.usagePercent‚76.4084948136119' disk.nvme1c1n1.out‚11405.5576171875! disk.dm-1.out‚5076.0537109375 disk.nvme0n1.in‚7.1875$ memory_percent‚96.98460102081299( proc.memory.availableMB‚ 494.04296875 network.sent‚ 5.756264e+06 disk.rbd1.in‚0 disk.md0.in‚0 disk.md0.out‚ 88.7734375" disk.nvme0n1.out‚ 18952.5546875 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.70329475402832 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚11405.5576171875 disk.dm-0.out‚25091.39453125 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 18952.5546875¾ Úâžô‚(:“ ©üöÒÔ’×Þ gpu.0.gpu‚95.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.33366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û]Io2õ- _wandbruntime‚3390  _runtime‚3390ØSÉ*ï&öéM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4165367773696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4159069955224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553676534517248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚550462855283650 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚3599 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚549510808272896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546303427727680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3390.324515289 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29048962 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4165709204992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123980306166784 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29048960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64602751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553676176046592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14885067 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚546303444767040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549510808272896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚549510825312256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4189403793408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550462497682904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14885355 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64612910 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6282678 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64602751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119791011829760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29048960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35553791 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8602677 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553676534517248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29048960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64602751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553676176046592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14885067 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚546303444767040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549510808272896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚549510825312256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4189403793408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550462497682904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14885355 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64612910 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6282678 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64602751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119791011829760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29048960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35553791 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8602677 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553676534517248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3390.324515289 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29048962 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528¦y©2ø- _wandbruntime‚3390  _runtime‚33904Þ:ùÿ 2loss/FBt2/assignment_loss‚0.028932164539739347 epoch‚8 trainer/global_step‚3599 _step‚295 1loss/SRqqt2/detection_loss‚0.9527446979662443 /loss/FBt2/detection_loss‚0.8662671808291365 0loss/FRt2/assignment_loss‚1.8748710253624001 1loss/SRqqt1/detection_loss‚0.6302425291440967 /loss/FBt1/detection_loss‚0.5203915344695171 $loss/total_loss‚8.92509448035381 $ _timestamp‚1.784528426057389e+09 0loss/FRt1/assignment_loss‚0.9354357696929068 2loss/SRqqt1/assignment_loss‚0.5267916899342773 2loss/SRqqt2/assignment_loss‚0.5409835671861459 /loss/FRt1/detection_loss‚1.4128936560711272 /loss/FRt2/detection_loss‚0.5836434541540597 _runtime‚ 3390.32532125 1loss/FBt1/assignment_loss‚0.051897211004160335„\Á ú„ 1loss/SRqqt2/detection_loss‚0.9527446979662443 /loss/FBt2/detection_loss‚0.8662671808291365 0loss/FRt2/assignment_loss‚1.8748710253624001 1loss/SRqqt1/detection_loss‚0.6302425291440967 /loss/FBt1/detection_loss‚0.5203915344695171 $loss/total_loss‚8.92509448035381 $ _timestamp‚1.784528426057389e+09 0loss/FRt1/assignment_loss‚0.9354357696929068 2loss/SRqqt1/assignment_loss‚0.5267916899342773 2loss/SRqqt2/assignment_loss‚0.5409835671861459 /loss/FRt1/detection_loss‚1.4128936560711272 /loss/FRt2/detection_loss‚0.5836434541540597 _runtime‚ 3390.32532125 1loss/FBt1/assignment_loss‚0.05189721100416033 2loss/FBt2/assignment_loss‚0.028932164539739347 epoch‚8 trainer/global_step‚3599 _step‚295§¤a–‡û‚(:ü ¸üöÒ»æÄÞ proc.cpu.threads‚17 network.sent‚ 5.806846e+06 network.recv‚890002) disk./.usagePercent‚76.39301264641281 disk.rbd1.out‚0 disk.md0.in‚0 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚18953.431640625 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚ 25092.015625 disk.dm-1.in‚0! disk.dm-1.out‚5076.2333984375& disk.nvme0c0n1.out‚18953.431640625$ memory_percent‚96.77150249481201( proc.memory.availableMB‚ 528.95703125% disk./.usageGB‚1172.8237266540527 disk.rbd1.in‚0% disk.nvme1n1.out‚11406.1728515625 disk.nvme0c0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 proc.memory.rssMB‚ 2864.65625 disk.md0.out‚ 89.09765625 disk.rbd0.in‚0' disk.nvme1c1n1.out‚11406.1728515625* proc.memory.percent‚17.484474182128906 cpu‚25.663586758558587S—ö†œü‚(:‘ ¸üöÒ£‚ß gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚279.053' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.01766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a’¨2ý- _wandbruntime‚3406  _runtime‚3406·¦ÍË=þÅ{ & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8335978388786316 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2046264261007309 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8034012913703918 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.666750967502594 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9820513129234314 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3098811209201813 *jet/accuracy_1_of_3‚0.9170092344284058 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37576547265052795 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.024948684498667717 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8467958569526672 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7369063496589661 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6373452544212341 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27491632103919983 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7359427213668823 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6883991956710815 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.034325484186410904 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08667805045843124 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.34523212909698486 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965208172798157 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4554084539413452 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28264036774635315 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9910256266593933 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6340882182121277 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2246713638305664 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 /particle/accuracy_2_of_2‚0.9998930096626282 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5749229192733765 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1337616890668869 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04589257016777992 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34264761209487915 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6392790675163269 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6699187159538269 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6778556108474731 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7951422333717346 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _step‚296 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.632819414138794 *jet/accuracy_2_of_3‚0.5667544603347778 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7257309556007385 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6401737928390503 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6060174703598022 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10369990766048431 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2335006445646286 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6444413065910339 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22008541226387024 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4453154504299164 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22161062061786652 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9902872443199158 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8387188911437988 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7314814329147339 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7468369007110596 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.927951455116272 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.031086402013897896 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03316069766879082 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.746008038520813 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚3599 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4367455840110779 +jet/accuracy_3_of_3‚0.07145556062459946 +jet/accuracy_3_of_4‚0.08376332372426987 'particle/f_score‚0.6462758779525757 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3300180733203888 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6520425081253052 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240449070930481 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8348639607429504 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049840934574604034 7validation_average_jet_accuracy‚0.38292020559310913 particle/accuracy_4_of_5‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5807047486305237 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8665940165519714 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27437740564346313 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.732250452041626 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8467958569526672 *jet/accuracy_1_of_2‚0.7857076525688171 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_2_of_5‚1 4 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9906005859375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5200902223587036 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚3406.952688382 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7427124977111816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_4‚0.7971013188362122 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.30844646692276 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5764106512069702 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.761014461517334 jet/accuracy_5_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8497294187545776 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7930592894554138 jet/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996330976486206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975225329399109 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09746217727661133 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37917762994766235 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8796376585960388 % _timestamp‚1.7845284426841335e+09 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2084716558456421 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984030723571777 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8298563361167908 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953126311302185 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7402786016464233 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7345805168151855 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2667851448059082 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5378841757774353 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6921594142913818 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28447869420051575 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4507758617401123 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956678748130798 ,jet/accuracy_4_of_4‚0.013479332439601421 /particle/accuracy_3_of_3‚0.9994933605194092 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46044376492500305 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9565331339836121 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚8 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21509644389152527 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 particle/accuracy_2_of_3‚1 )loss/val_total_loss‚9.738765159744881 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20667608082294464 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28353384137153625 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_4_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_3_of_4‚0.9968253374099731 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.519928514957428 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6781687140464783 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03067934140563011 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976717829704285 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032559603452682495 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8691442608833313 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6807916760444641 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20600545406341553 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22264814376831055 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948258399963379 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6373452544212341 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19989299774169922 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11914129555225372 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.9072805047035217 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7064949870109558 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917100071907043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 *jet/accuracy_1_of_4‚0.9643596410751343 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1174406185746193 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958595037460327 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5823105573654175 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960717558860779 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7373017072677612 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7314814329147339 jet/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4687928557395935 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2349829375743866 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.044285986572504044 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6255258917808533 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7551976442337036 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26733478903770447 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 )jet/accuracy_2_of_2‚0.152878999710083 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28456220030784607 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7338635921478271 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6195027232170105 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949597716331482 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27491632103919983 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 particle/accuracy_1_of_4‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.024948684498667717 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293ærüÑ,ÿÊ{ 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7314814329147339 jet/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4687928557395935 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2349829375743866 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.044285986572504044 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6255258917808533 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7551976442337036 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26733478903770447 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 )jet/accuracy_2_of_2‚0.152878999710083 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28456220030784607 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7338635921478271 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6195027232170105 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949597716331482 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27491632103919983 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 particle/accuracy_1_of_4‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.024948684498667717 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8335978388786316 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2046264261007309 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8034012913703918 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.666750967502594 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9820513129234314 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3098811209201813 *jet/accuracy_1_of_3‚0.9170092344284058 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37576547265052795 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.024948684498667717 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8467958569526672 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7369063496589661 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6373452544212341 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27491632103919983 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7359427213668823 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6883991956710815 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.034325484186410904 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08667805045843124 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.34523212909698486 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965208172798157 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4554084539413452 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28264036774635315 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9910256266593933 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6340882182121277 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2246713638305664 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 /particle/accuracy_2_of_2‚0.9998930096626282 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5749229192733765 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1337616890668869 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04589257016777992 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34264761209487915 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6392790675163269 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6699187159538269 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6778556108474731 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7951422333717346 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _step‚296 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.632819414138794 *jet/accuracy_2_of_3‚0.5667544603347778 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7257309556007385 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6401737928390503 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6060174703598022 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10369990766048431 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2335006445646286 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6444413065910339 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22008541226387024 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4453154504299164 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22161062061786652 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9902872443199158 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8387188911437988 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7314814329147339 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7468369007110596 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.927951455116272 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.031086402013897896 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03316069766879082 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.746008038520813 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚3599 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4367455840110779 +jet/accuracy_3_of_3‚0.07145556062459946 +jet/accuracy_3_of_4‚0.08376332372426987 'particle/f_score‚0.6462758779525757 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3300180733203888 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6520425081253052 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240449070930481 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8348639607429504 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049840934574604034 7validation_average_jet_accuracy‚0.38292020559310913 particle/accuracy_4_of_5‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5807047486305237 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8665940165519714 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27437740564346313 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.732250452041626 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8467958569526672 *jet/accuracy_1_of_2‚0.7857076525688171 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_2_of_5‚1 4 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9906005859375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5200902223587036 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚3406.952688382 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7427124977111816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_4‚0.7971013188362122 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.30844646692276 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5764106512069702 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.761014461517334 jet/accuracy_5_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8497294187545776 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7930592894554138 jet/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996330976486206 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975225329399109 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09746217727661133 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37917762994766235 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8796376585960388 % _timestamp‚1.7845284426841335e+09 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2084716558456421 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984030723571777 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8298563361167908 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953126311302185 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7402786016464233 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7345805168151855 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2667851448059082 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5378841757774353 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/evº`ÿent_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6921594142913818 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28447869420051575 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4507758617401123 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956678748130798 ,jet/accuracy_4_of_4‚0.013479332439601421 /particle/accuracy_3_of_3‚0.9994933605194092 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46044376492500305 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9565331339836121 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚8 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21509644389152527 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 particle/accuracy_2_of_3‚1 )loss/val_total_loss‚9.738765159744881 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20667608082294464 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28353384137153625 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_4_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_3_of_4‚0.9968253374099731 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.519928514957428 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6781687140464783 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03067934140563011 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976717829704285 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032559603452682495 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8691442608833313 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6807916760444641 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20600545406341553 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22264814376831055 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948258399963379 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6373452544212341 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19989299774169922 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11914129555225372 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.9072805047035217 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7064949870109558 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917100071907043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 *jet/accuracy_1_of_4‚0.9643596410751343 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1174406185746193 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958595037460327 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5823105573654175 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960717558860779 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7373017072677612¨š¤Óv€‚(:ë ÇüöÒ’…©Þ proc.memory.rssMB‚ 2903.1875% disk./.usageGB‚1172.8236846923828" disk.dm-0.out‚25096.4794921875 disk.nvme0c0n1.in‚7.1875# disk.nvme0c0n1.out‚ 18957.953125 disk.rbd1.in‚0 disk.rbd1.out‚0& proc.memory.availableMB‚ 737.828125 cpu‚25.663530829740434 network.sent‚ 5.858806e+06 network.recv‚894249 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.nvme1n1.in‚ 1.45703125* proc.memory.percent‚17.719650268554688 proc.cpu.threads‚17 disk.dm-0.in‚ 8.64453125! disk.dm-1.out‚5321.6279296875 disk.rbd0.in‚0 disk.md0.out‚ 89.41015625$ memory_percent‚95.49665451049805) disk./.usagePercent‚76.39300991319868! disk.nvme0n1.out‚ 18957.953125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125! disk.nvme1c1n1.out‚ 11652.1875 disk.nvme1n1.out‚ 11652.1875 disk.md0.in‚0úU#ø“‚(:ˆ ÇüöÒò¾åÞ gpu.0.gpu‚61.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.018' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªïíׇ‚‚(:ü ÖüöÒ×£ÿÝ disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875& proc.memory.availableMB‚ 737.828125 disk.md0.out‚ 89.703125 disk.rbd0.out‚0( disk.nvme1c1n1.out‚11716.71923828125 disk.rbd1.in‚0& disk.nvme1n1.out‚11716.71923828125$ memory_percent‚95.49665451049805 proc.cpu.threads‚17% disk.nvme0n1.out‚18958.8916015625 network.recv‚897837$ disk./.usageGB‚1172.825439453125 disk.dm-0.in‚ 8.64453125' disk.nvme0c0n1.out‚18958.8916015625 disk.nvme1n1.in‚ 1.45703125* proc.memory.percent‚17.719650268554688 cpu‚25.6626079167794( disk./.usagePercent‚76.3931242112441 disk.md0.in‚0" disk.dm-0.out‚25097.1708984375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 proc.memory.rssMB‚ 2903.1875 network.sent‚ 5.902998e+06" disk.dm-1.out‚5385.76416015625ÂV`¹œƒ‚(:‘ ÖüöÒ°‰ÔÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.902' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.96733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ°ûJ‹„‚(:€ åüöÒŒòùÝ( proc.memory.availableMB‚ 737.80078125 cpu‚25.661832789973474 network.sent‚ 5.909763e+06 network.recv‚901310 disk.rbd1.out‚0& disk.nvme1n1.out‚11900.24365234375 disk.md0.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1173.1273002624512 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125" disk.dm-0.out‚25164.0107421875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875& disk.nvme0c0n1.out‚19024.498046875$ disk.nvme0n1.out‚19024.498046875( disk./.usagePercent‚76.4127862082691 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.out‚ 90.0859375" disk.dm-1.out‚5567.23291015625 disk.rbd0.in‚0$ memory_percent‚95.49682140350342 proc.memory.rssMB‚ 2903.1875* proc.memory.percent‚17.719650268554688( disk.nvme1c1n1.out‚11900.24365234375 disk.dm-0.in‚ 8.64453125'‘ïž…‚(:“ åüöÒÖˆÈÞ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚260.889' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.96300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Tº2†- _wandbruntime‚3452  _runtime‚3452/툱ȇ ' lr-AdamW/pg2‚0.00016693063519613723 trainer/global_step‚3649 $ _timestamp‚1.784528487810982e+09 _runtime‚3452.079115458 _step‚297 ' lr-AdamW/pg1‚0.00016693063519613723•=CrÍˆÇ $ _timestamp‚1.784528487810982e+09 _runtime‚3452.079115458 _step‚297 ' lr-AdamW/pg1‚0.00016693063519613723 ' lr-AdamW/pg2‚0.00016693063519613723 trainer/global_step‚3649©ãÙ2‰- _wandbruntime‚3452  _runtime‚3452ÉI)Hä'ŠÞO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4240486039040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65663911 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562958747368448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4264853426176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 % _timestamp‚1.7845284878120055e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4264752351232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36110286 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4240144614912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29553625 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562959105831936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6371760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29553627 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚3649 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36120429 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8749343 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚558718619792896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65674056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559682535071314 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29553627 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15121388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121439442338304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4240486039040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4240144614912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8749628 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555448803481392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36120429 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125704295764480 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6371760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29553627 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚3649 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36120429 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8749343 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚558718619792896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65674056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559682535071314 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29553627 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15121388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121439442338304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4240486039040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4240144614912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8749628 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555448803481392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36120429 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125704295764480 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562958747368448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4264853426176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 % _timestamp‚1.7845284878120055e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4264752351232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36110286 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4240144614912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29553625 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562959105831936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344ª—rÞ2Œ- _wandbruntime‚3452  _runtime‚3452ô¾NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15125325 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36119942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29561418 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36130099 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558866877255680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29561418 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 trainer/global_step‚3649 _runtime‚3452.710603517 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.curre°p*q¡nt‚291 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36130099 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29561420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 # _timestamp‚1.78452848844166e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚563108490440192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15125034 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558866877255680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4241596145152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555596234148864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65681360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29561420 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8751543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4265872225792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555596217109504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125738476221952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4235197578920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚558866894295040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559831074127038 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36119942 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65691519 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8751831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125738591707648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚558866894295040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4241254713856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4265969099264ƒíoñ&ŽëM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65681360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29561420 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8751543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4265872225792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555596217109504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125738476221952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4235197578920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚558866894295040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559831074127038 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36119942 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65691519 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8751831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125738591707648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚558866894295040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4241254713856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4265969099264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚559831431727784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15125325 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36119942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29561418 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36130099 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558866877255680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29561418 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 trainer/global_step‚3649 _runtime‚3452.710603517 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36130099 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29561420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 # _timestamp‚1.78452848844166e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚563108490440192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15125034 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558866877255680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4241596145152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555596234148864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184« àë”2- _wandbruntime‚3452  _runtime‚3452·ž. %loss/total_loss‚8.941334268769467 1loss/SRqqt1/assignment_loss‚0.610337409199784 3loss/FBt2/assignment_loss‚0.0005023962143296052 /loss/FRt1/detection_loss‚1.2867865877410825 /loss/FRt2/detection_loss‚0.5742361704681128 /loss/FBt1/detection_loss‚0.5172721985102033 _runtime‚3452.711721859 _step‚300 2loss/SRqqt2/assignment_loss‚0.6819912925049201 2loss/FBt1/assignment_loss‚0.035488748889940205 0loss/SRqqt2/detection_loss‚0.945432324512777 epoch‚9 0loss/FRt1/assignment_loss‚0.7888387148779941 0loss/FRt2/assignment_loss‚1.9531141930699474 0loss/SRqqt1/detection_loss‚0.629889646222598 % _timestamp‚1.7845284884434283e+09 /loss/FBt2/detection_loss‚0.9174445865577773 trainer/global_step‚3649ƒý¥ ‘† 0loss/FRt1/assignment_loss‚0.7888387148779941 0loss/FRt2/assignment_loss‚1.9531141930699474 0loss/SRqqt1/detection_loss‚0.629889646222598 % _timestamp‚1.7845284884434283e+09 /loss/FBt2/detection_loss‚0.9174445865577773 trainer/global_step‚3649 %loss/total_loss‚8.941334268769467 1loss/SRqqt1/assignment_loss‚0.610337409199784 3loss/FBt2/assignment_loss‚0.0005023962143296052 /loss/FRt1/detection_loss‚1.2867865877410825 /loss/FRt2/detection_loss‚0.5742361704681128 /loss/FBt1/detection_loss‚0.5172721985102033 _runtime‚3452.711721859 _step‚300 2loss/SRqqt2/assignment_loss‚0.6819912925049201 2loss/FBt1/assignment_loss‚0.035488748889940205 0loss/SRqqt2/detection_loss‚0.945432324512777 epoch‚9¬i¯]¶ƒ’‚(:ø ôüöÒ“¿±Þ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.md0.out‚ 90.49609375" disk.dm-0.out‚25295.2763671875% disk.nvme0c0n1.out‚19120.22265625 disk.nvme1c1n1.in‚ 1.45703125# disk.nvme1n1.out‚11936.82421875$ memory_percent‚95.49705982208252( proc.memory.availableMB‚ 737.76171875 proc.memory.rssMB‚ 2903.1875! disk.dm-1.out‚5567.3935546875 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 cpu‚25.660913348055693 network.sent‚ 5.92491e+06% disk./.usageGB‚1173.0631790161133 disk.dm-1.in‚0% disk.nvme1c1n1.out‚11936.82421875* proc.memory.percent‚17.719650268554688 proc.cpu.threads‚17 network.recv‚904059) disk./.usagePercent‚76.40860960860533 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125# disk.nvme0n1.out‚19120.22265625Sâ Ð“‚(:’ ôüöÒׂðÞ gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.03' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.00999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÉD@‹”‚(:€ ƒýöÒ‰«Þ! disk.dm-1.out‚5567.4169921875 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 network.recv‚908460 disk.rbd1.out‚0& disk.nvme0n1.out‚19181.48876953125 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 738.0078125* proc.memory.percent‚17.719650268554688 proc.cpu.threads‚17 disk.md0.out‚ 90.7890625 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚25356.47119140625 cpu‚25.65986626354411) disk./.usagePercent‚76.40864017090877$ disk./.usageGB‚1173.063648223877 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125( disk.nvme0c0n1.out‚19181.48876953125$ memory_percent‚95.49555778503418 proc.memory.rssMB‚ 2903.1875 network.sent‚ 5.976441e+06' disk.nvme1c1n1.out‚11937.4267578125% disk.nvme1n1.out‚11937.4267578125 disk.md0.in‚0 disk.dm-1.in‚0h$D¯“•‚(:ˆ ƒýöÒžŠõÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚218.178' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚72.726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Ø‹^„–‚(:ù ’ýöÒÕ§‘Þ# disk.dm-0.out‚25357.30712890625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.md0.in‚0( disk.nvme0c0n1.out‚19182.46044921875 disk.rbd1.out‚0 disk.nvme1n1.out‚ 11938.34375$ memory_percent‚95.49558162689209 proc.memory.rssMB‚ 2903.1875 proc.cpu.threads‚17! disk.dm-1.out‚5567.5888671875 disk.nvme0n1.in‚7.1875 network.sent‚ 5.979405e+06 network.recv‚911923 disk.dm-0.in‚ 8.64453125 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 disk.rbd0.out‚0" disk.nvme1c1n1.out‚ 11938.34375( proc.memory.availableMB‚ 738.00390625* proc.memory.percent‚17.719650268554688 cpu‚25.659044028856794 disk.nvme0c0n1.in‚7.1875& disk.nvme0n1.out‚19182.46044921875) disk./.usagePercent‚76.40865980035571$ disk./.usageGB‚1173.063949584961 disk.md0.out‚ 91.19921875™,°»ž—‚(:“ ’ýöÒÂîâÞ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.168' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.38933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü›`Ö2˜- _wandbruntime‚3497  _runtime‚3497Ðç¥gəà _runtime‚3497.522485862 _step‚301 ' lr-AdamW/pg1‚0.00016657170868903602 ' lr-AdamW/pg2‚0.00016657170868903602 trainer/global_step‚3699 % _timestamp‚1.7845285332538393e+09nš¾ÎšÈ trainer/global_step‚3699 % _timestamp‚1.7845285332538393e+09 _runtime‚3497.522485862 _step‚301 ' lr-AdamW/pg1‚0.00016657170868903602 ' lr-AdamW/pg2‚0.00016657170868903602­¨ìùœ2›- _wandbruntime‚3497  _runtime‚3497ü*Üd^'œßO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127419131169280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚3699 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4296055140864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36594938 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4295713716736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15322898 _step‚302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6458460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚566132344900096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚562820319815632 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570428041577472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4296055140864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4320693939200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚570428400040960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚566132327860736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127419257147904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123098437230080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4320801305600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚566132344900096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566132327860736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚562820336854992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29943275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 % _timestamp‚1.7845285332545989e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4295713716736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66548358 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36594938 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8864438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4289231842690 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66538213 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚570428400040960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29943277 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚567109909258956 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36605081 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚567109551658322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29943277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8864153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570428041577472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4296055140864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4320693939200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚570428400040960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚566132327860736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127419257147904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123098437230080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4320801305600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚566132344900096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566132327860736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚562820336854992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29943275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 % _timestamp‚1.7845285332545989e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4295713716736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66548358 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36594938 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8864438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4289231842690 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66538213 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚570428400040960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29943277 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚567109909258956 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36605081 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚567109551658322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29943277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8864153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15326632 trainer/global_step‚3699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29951070 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6460194 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4297165246976 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚567258448314792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4321914435072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36604594 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123131617500160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚562967750483104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845285338895462e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚566280619402240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29951070 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8866724 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚570577784649216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8866441 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123131636112384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4297165246976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66555662 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127453430867456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15326918 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36604594 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚567258090714046 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚303 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚570577426178560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29951068 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36614751 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚562967767522464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4290340230942 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29951068 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6460191 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66565821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚570577784649216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4290680792328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66565821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566280602362880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚566280602362880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ãC±ô& îM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚570577426178560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29951068 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36614751 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚562967767522464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4290340230942 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29951068 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6460191 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66565821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚570577784649216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4290680792328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66565821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566280602362880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚566280602362880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4296823815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15326632 trainer/global_step‚3699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29951070 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6460194 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4297165246976 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚567258448314792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4321914435072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36604594 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123131617500160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚562967750483104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845285338895462e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚566280619402240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29951070 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8866724 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚570577784649216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8866441 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123131636112384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4297165246976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66555662 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127453430867456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15326918 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36604594 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚567258090714046 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚303¯§ÖNf2¡- _wandbruntime‚3498  _runtime‚3498°pÌ¢ 0loss/FRt1/assignment_loss‚0.8678368021866043 1loss/SRqqt1/assignment_loss‚0.510070306193517 1loss/FBt1/assignment_loss‚0.02470448783170367 .loss/FRt1/detection_loss‚1.474301206336511 /loss/FRt2/detection_loss‚0.5284782867001099 /loss/FBt1/detection_loss‚0.8956443749881517 2loss/SRqqt2/assignment_loss‚0.6231499659478642 3loss/FBt2/assignment_loss‚0.0006629957606018403 1loss/SRqqt2/detection_loss‚1.0209012705542635 /loss/FBt2/detection_loss‚1.0730998596797388 trainer/global_step‚3699 % _timestamp‚1.7845285338913126e+09 _step‚304 0loss/FRt2/assignment_loss‚1.9982910996618963 1loss/SRqqt1/detection_loss‚0.5884708524665231 %loss/total_loss‚9.605611508307486 epoch‚9 _runtime‚3498.159473143yC £† 0loss/FRt1/assignment_loss‚0.8678368021866043 1loss/SRqqt1/assignment_loss‚0.510070306193517 1loss/FBt1/assignment_loss‚0.02470448783170367 .loss/FRt1/detection_loss‚1.474301206336511 /loss/FRt2/detection_loss‚0.5284782867001099 /loss/FBt1/detection_loss‚0.8956443749881517 2loss/SRqqt2/assignment_loss‚0.6231499659478642 3loss/FBt2/assignment_loss‚0.0006629957606018403 1loss/SRqqt2/detection_loss‚1.0209012705542635 /loss/FBt2/detection_loss‚1.0730998596797388 trainer/global_step‚3699 % _timestamp‚1.7845285338913126e+09 _step‚304 0loss/FRt2/assignment_loss‚1.9982910996618963 1loss/SRqqt1/detection_loss‚0.5884708524665231 %loss/total_loss‚9.605611508307486 epoch‚9 _runtime‚3498.159473143°q ãö¤‚(:û ¡ýöÒ–©ýÞ$ memory_percent‚95.49751281738281 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚11996.1474609375 disk.md0.in‚0" disk.dm-0.out‚25425.6259765625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚19193.8134765625% disk.nvme0n1.out‚19193.8134765625 cpu‚25.65899274289702 network.recv‚915549) disk./.usagePercent‚76.42834565088582% disk./.usageGB‚1173.3661766052246 disk.rbd0.in‚0' disk.nvme1c1n1.out‚11996.1474609375 disk.dm-0.in‚ 8.64453125$ proc.memory.availableMB‚737.6875* proc.memory.percent‚17.719650268554688! disk.dm-1.out‚5567.7646484375 disk.nvme0n1.in‚7.1875 proc.memory.rssMB‚ 2903.1875 proc.cpu.threads‚17 network.sent‚ 5.998231e+06 disk.rbm…%-d0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.out‚ 91.4921875 disk.nvme0c0n1.in‚7.1875£fœ¥‚(:‘ ¡ýöÒÅÈåÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.94133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NvEž¦‚(:“ °ýöÒÐΠÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚123.025' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.00833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü³:}§‚(:ò °ýöÒµ­ÁÞ disk.dm-1.in‚0" disk.nvme0c0n1.out‚ 19347.46875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚12031.56201171875 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 6.046663e+06 network.recv‚919636 disk.dm-0.in‚ 8.64453125 disk.nvme0n1.in‚7.1875 disk.rbd1.out‚0# memory_percent‚95.4986572265625! proc.memory.availableMB‚737.5 proc.memory.rssMB‚ 2903.1875) disk./.usagePercent‚76.42417476612434% disk./.usageGB‚1173.3021430969238 disk.md0.in‚0 disk.nvme0n1.out‚ 19347.46875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚12031.56201171875 cpu‚25.660232435718182 disk.md0.out‚ 91.83984375" disk.dm-0.out‚25613.7392578125" disk.dm-1.out‚5567.96533203125 disk.nvme0c0n1.in‚7.1875* proc.memory.percent‚17.719650268554688ÿšÏ†¨‚(:û ¿ýöÒø‘ýÝ proc.cpu.threads‚17 network.sent‚ 6.05349e+06" disk.dm-0.out‚25617.9326171875 disk.dm-1.in‚0" disk.dm-1.out‚5568.02001953125 disk.nvme0c0n1.in‚7.1875% disk.nvme0c0n1.out‚19351.57421875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 network.recv‚923161$ disk./.usageGB‚1173.302043914795 disk.dm-0.in‚ 8.64453125 disk.rbd1.out‚0& disk.nvme1n1.out‚12032.37255859375& proc.memory.availableMB‚ 737.734375 proc.memory.rssMB‚ 2903.1875) disk./.usagePercent‚76.42416830580002 disk.md0.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 disk.rbd1.in‚0 cpu‚25.661884448960237 disk.md0.out‚ 92.1484375# disk.nvme0n1.out‚19351.57421875( disk.nvme1c1n1.out‚12032.37255859375 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.49722671508789* proc.memory.percent‚17.719650268554688BZfÞ“©‚(:ˆ ¿ýöÒ¹ýÁÞ gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚239.964' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚79.988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tgÅ2ª- _wandbruntime‚3541  _runtime‚3541ãç É«Ã _runtime‚3541.432194942 _step‚305 ' lr-AdamW/pg1‚0.00016620782495779641 ' lr-AdamW/pg2‚0.00016620782495779641 trainer/global_step‚3749 % _timestamp‚1.7845285771639905e+09_ê![Î¬È % _timestamp‚1.7845285771639905e+09 _runtime‚3541.432194942 _step‚305 ' lr-AdamW/pg1‚0.00016620782495779641 ' lr-AdamW/pg2‚0.00016620782495779641 trainer/global_step‚3749±¦¯2­- _wandbruntime‚3541  _runtime‚3541s©Òä'®ÞO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573546070007296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4351282818560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15523507 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37079590 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8978350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30332927 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784528577164793e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6545160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37089733 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573546052967936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37089733 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37079590 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129134005576192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15523796 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30332927 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4351282818560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚574536568245330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124757450734080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4344715056098 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129134135749120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4351624242688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8978636 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚577897694249984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚3541.436603495 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67412515 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577897335786496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15523507 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37079590 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8978350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30332927 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784528577164793e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6545160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37089733 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573546052967936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37089733 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37079590 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129134005576192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15523796 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30332927 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4351282818560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚574536568245330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124757450734080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4344715056098 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129134135749120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4351624242688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8978636 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚577897694249984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚3541.436603495 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67412515 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577897335786496 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578046720387584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67440123 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6546891 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352734348800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚573694327470080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352734348800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15527578 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124790631004160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚574685107301054 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578047078858240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4346164005736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67440123 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37089246 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8980687 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30340720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129168430599168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30340720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37099403 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30340718 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6546894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352392917504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4377669429248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573694327470080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352392917504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15527867 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124790649616384 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647Ùk^/ó&²íM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚573694327470080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352734348800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15527578 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124790631004160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚574685107301054 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578047078858240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4346164005736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67440123 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37089246 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8980687 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30340720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129168430599168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30340720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37099403 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30340718 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6546894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352392917504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4377669429248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573694327470080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352392917504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15527867 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124790649616384 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578046720387584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67440123 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6546891 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352734348800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126416482850304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4407193344512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6631857 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15724649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130848976828416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚580959795114496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚585366629995520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7845286211652892e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30722575 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130848857141248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37574385 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37574385 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9092792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚581963942432972 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4400198269506 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68296962 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚585366988459008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30722577 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚585366629995520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37564242 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚580959795114496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577563403602192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _step‚310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚580959778075136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9093075 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4432493978112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚3799 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126416482850304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4407193344512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6631857 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15724649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130848976828416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚580959795114496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚585366629995520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7845286211652892e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30722575 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130848857141248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37574385 TGDeviceStatsMonitor.on_train_batch_start/all{iMz‰ocation.small_pool.allocated‚37574385 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9092792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚581963942432972 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4400198269506 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68296962 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚585366988459008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30722577 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚585366629995520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37564242¶¾Þ§2Â- _wandbruntime‚3586  _runtime‚3586Ë©Ù³ï&ÃéM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚585516014596608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6633594 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585516373067264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4433613642752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4408303450624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9095256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37584055 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚585516373067264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433510477824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚3586.071926882 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37584055 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126449644508160 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68314425 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582112481488808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6633591 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126449663120384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30730368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577710834269664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15728558 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68314425 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚581108052577280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15728850 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37573898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30730370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30730368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4407962019328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4407962019328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4401306657758 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚3799 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37584055 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚585516373067264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433510477824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚3586.071926882 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37584055 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126449644508160 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68314425 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582112481488808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6633591 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126449663120384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30730368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577710834269664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15728558 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68314425 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚581108052577280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15728850 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37573898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30730370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30730368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4407962019328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4407962019328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4401306657758 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚3799 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚585516014596608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272·ì´½2Å- _wandbruntime‚3586  _runtime‚3586Íý:c ƃ epoch‚9 _step‚312 0loss/FRt1/assignment_loss‚0.7927620555451494 2loss/SRqqt1/assignment_loss‚0.5588072545955381 3loss/SRqqt2/assignment_loss‚0.49488780645401437 /loss/FRt2/detection_loss‚0.5110539508864977 1loss/SRqqt1/detection_loss‚0.5937615207797434 1loss/SRqqt2/detection_loss‚1.0644210754155752 trainer/global_step‚3799 0loss/FRt2/assignment_loss‚1.9354397540654593 /loss/FBt2/detection_loss‚0.7704841469359749 %loss/total_loss‚8.707231349315693 _runtime‚3586.076600071 1loss/FBt1/assignment_loss‚0.05563238652251347 .loss/FRt1/detection_loss‚1.397449698998128 % _timestamp‚1.7845286218047645e+09 3loss/FBt2/assignment_loss‚0.0010191519225235665 /loss/FBt1/detection_loss‚0.5315125471945757ÔžïLj 3loss/FBt2/assignment_loss‚0.0010191519225235665 /loss/FBt1/detection_loss‚0.5315125471945757 epoch‚9 _step‚312 0loss/FRt1/assignment_loss‚0.7927620555451494 2loss/SRqqt1/assignment_loss‚0.5588072545955381 3loss/SRqqt2/assignment_loss‚0.49488780645401437 /loss/FRt2/detection_loss‚0.5110539508864977 1loss/SRqqt1/detection_loss‚0.5937615207797434 1loss/SRqqt2/detection_loss‚1.0644210754155752 trainer/global_step‚3799 0loss/FRt2/assignment_loss‚1.9354397540654593 /loss/FBt2/detection_loss‚0.7704841469359749 %loss/total_loss‚8.707231349315693 _runtime‚3586.076600071 1loss/FBt1/assignment_loss‚0.05563238652251347 .loss/FRt1/detection_loss‚1.397449698998128 % _timestamp‚1.7845286218047645e+09¸Ð\^ˆ‰È‚(:þ ûýöÒŸ§ÄÞ network.sent‚ 6.145217e+06) disk./.usagePercent‚76.42410569034907 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.out‚25631.41796875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875 network.recv‚944435 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1n1.out‚12036.6611328125 disk.md0.in‚0 disk.md0.out‚ 93.48828125' disk.nvme0c0n1.out‚19364.1962890625 disk.nvme0n1.in‚7.1875% disk.nvme0n1.out‚19364.1962890625' disk.nvme1c1n1.out‚12036.6611328125 disk.dm-0.in‚ 8.64453125 disk.dm-1.out‚ 5568.546875 disk.rbd0.in‚0( proc.memory.availableMB‚ 811.87890625$ disk./.usageGB‚1173.301082611084 disk.rbd0.out‚0$ memory_percent‚95.04468441009521# proc.memory.rssMB‚ 2823.62890625* proc.memory.percent‚17.234063148498535 cpu‚25.666448976890898 proc.cpu.threads‚17ƒ< &œÉ‚(:‘ ûýöÒϸÈÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.118' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.03933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌË·ŽÊ‚(:ƒ ŠþöÒÚÛäÝ disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.04344463348389 disk.md0.in‚0 disk.md0.out‚ 93.74609375 disk.dm-0.in‚ 8.64453125 disk.rbd0.in‚0& disk.nvme1n1.out‚12037.14501953125# proc.memory.rssMB‚ 2823.62890625* proc.memory.percent‚17.234063148498535 proc.cpu.threads‚17) disk./.usagePercent‚76.42411811404966 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0( disk.nvme1c1n1.out‚12037.14501953125 disk.rbd1.out‚0( proc.memory.availableMB‚ 812.08203125 network.sent‚ 6.194888e+06 network.recv‚948326% disk./.usageGB‚1173.3012733459473" disk.dm-0.out‚25632.9501953125 disk.nvme0c0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 cpu‚25.66825090292153 disk.dm-1.in‚0" disk.dm-1.out‚5568.65380859375% disk.nvme0c0n1.out‚19365.92578125# disk.nvme0n1.out‚19365.92578125÷+ÊË“Ë‚(:ˆ ŠþöÒãø¾Þ gpu.0.gpu‚41.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.043' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.681 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y }ŠÌ‚(:ÿ ™þöÒÔ„¦Þ) disk./.usagePercent‚76.42411562930953% disk./.usageGB‚1173.3012351989746 disk.rbd1.out‚0 disk.dm-0.out‚ 25635.3359375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚19368.31689453125 disk.rbd0.in‚0 network.sent‚ 6.201537e+06 network.recv‚952115 disk.rbd1.in‚0$ disk.nvme1n1.out‚12037.693359375 disk.md0.in‚0 disk.md0.out‚ 93.97265625 disk.nvme1c1n1.in‚ 1.45703125& disk.nvme1c1n1.out‚12037.693359375" disk.dm-1.out‚5568.68896484375 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚19368.31689453125 disk.rbd0.out‚0# memory_percent‚95.0434684753418& proc.memory.availableMB‚ 812.078125* proc.memory.percent‚17.234086990356445 cpu‚25.669539535091193 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125" proc.memory.rssMB‚ 2823.6328125 proc.cpu.threads‚17õ€Í‚(:’ ™þöÒöÎáÞ gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.08' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.02666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ̯2Î- _wandbruntime‚3631  _runtime‚3631æˆÆÈÏ ' lr-AdamW/pg2‚0.00016546527962608233 trainer/global_step‚3849 $ _timestamp‚1.784528667090012e+09 _runtime‚3631.358187018 _step‚313 ' lr-AdamW/pg1‚0.00016546527962608233ÞÒÍ­ÍÐÇ ' lr-AdamW/pg1‚0.00016546527962608233 ' lr-AdamW/pg2‚0.00016546527962608233 trainer/global_step‚3849 $ _timestamp‚1.784528667090012e+09 _runtime‚3631.358187018 _step‚313¹igþŒ2Ñ- _wandbruntime‚3631  _runtime‚36311 .5ÒÝO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592836282668032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4462762446336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69171264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38048894 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31112225 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9206366 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128075496354304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚592836282668032 `LDeŸ–—®"viceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚584934936975792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9206084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588373520221696 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38059037 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132563692921344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4462421022208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚3849 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31112227 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4456022044188 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592835924204544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 3631.35922796 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69161119 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚314 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592835924204544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31112227 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4462421022208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455681482914 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588373503182336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845286670910883e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38048894 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588373520221696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15924926ä-çPè'ÓâO trainer/global_step‚3849 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31112227 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4456022044188 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592835924204544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 3631.35922796 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69161119 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚314 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592835924204544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31112227 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4462421022208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455681482914 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588373503182336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845286670910883e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38048894 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588373520221696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15924926 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592836282668032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4462762446336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69171264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38048894 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31112225 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9206366 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128075496354304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚592836282668032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚584934936975792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9206084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588373520221696 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38059037 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132563692921344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4462421022208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180ºõþUS2Ô- _wandbruntime‚3631  _runtime‚3631Ô[ýï&ÕéM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38068707 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚3849 _step‚315 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38058550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3631.990514301 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589539140475070 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31120018 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4463872552448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15928678 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456789871166 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128108676624384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6720291 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132597986760704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9208387 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚588521777684480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31120018 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6720294 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128108658012160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592985667276288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚592985308805632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4463531121152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15928952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31120020 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69188727 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4489431913472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69188727 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589539498075816 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 % _timestamp‚1.7845286677222095e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚588521777684480 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38068707 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4457130432552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31120018 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4463872552448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15928678 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456789871166 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128108676624384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6720291 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132597986760704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9208387 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚588521777684480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31120018 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6720294 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128108658012160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592985667276288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚592985308805632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4463531121152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15928952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31120020 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69188727 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4489431913472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69188727 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589539498075816 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 % _timestamp‚1.7845286677222095e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚588521777684480 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38068707 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4457130432552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚3849 _step‚315 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38058550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3631.990514301»‰Ì2×- _wandbruntime‚3631  _runtime‚3631꽄Øÿ 1loss/FBt2/assignment_loss‚0.17209009629074026 /loss/FRt1/detection_loss‚1.3754717316421818 .loss/FRt2/detection_loss‚0.576062682216437 0loss/SRqqt1/detection_loss‚0.607435555057219 epoch‚9 2loss/SRqqt1/assignment_loss‚0.5527523753197241 2loss/FBt1/assignment_loss‚0.047199230490091074 1loss/SRqqt2/detection_loss‚0.9955341978694159 .loss/FBt2/detection_loss‚1.565447700326331 % _timestamp‚1.7845286677233057e+09 2loss/SRqqt2/assignment_loss‚0.5774541197792639 trainer/global_step‚3849 _runtime‚3631.991150992 _step‚316 0loss/FRt2/assignment_loss‚1.8580635668190777 /loss/FBt1/detection_loss‚0.4558430064978259 %loss/total_loss‚9.412924489915817 0loss/FRt1/assignment_loss‚0.6295702276075104Öð½) Ù„ % _timestamp‚1.7845286677233057e+09 2loss/SRqqt2/assignment_loss‚0.5774541197792639 trainer/global_step‚3849 _runtime‚3631.991150992 _step‚316 0loss/FRt2/assignment_loss‚1.8580635668190777 /loss/FBt1/detection_loss‚0.4558430064978259 %loss/total_loss‚9.412924489915817 0loss/FRt1/assignment_loss‚0.6295702276075104 1loss/FBt2/assignment_loss‚0.17209009629074026 /loss/FRt1/detection_loss‚1.3754717316421818 .loss/FRt2/detection_loss‚0.576062682216437 0loss/SRqqt1/detection_loss‚0.607435555057219 epoch‚9 2loss/SRqqt1/assignment_loss‚0.5527523753197241 2loss/FBt1/assignment_loss‚0.047199230490091074 1loss/SRqqt2/detection_loss‚0.9955341978694159 .loss/FBt2/detection_loss‚1.565447700326331¼á)¹‰Ú‚(:þ ¨þöң׺Þ% disk.nvme0n1.out‚19369.0302734375% disk.nvme1n1.out‚12038.1279296875$ memory_percent‚95.04196643829346) disk./.usagePercent‚76.42405375928061' disk.nvme0c0n1.out‚19369.0302734375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚12038.1279296875# proc.memory.rssMB‚ 2823.63671875* proc.memory.percent‚17.234110832214355 network.sent‚ 6.221254e+06 disk.md0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0( proc.memory.availableMB‚ 812.32421875 cpu‚25.671164148883474 proc.cpu.threads‚17% disk./.usageGB‚1173.3002853393555 disk.dm-0.out‚25635.89453125 disk.dm-1.out‚ 5568.8203125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125 network.recv‚961868 disk.md0.out‚ 94.171875 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0—kÛ‚(:’ ¨þöÒ綴Þ gpu.0.gpu‚45.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.53' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.50999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤,u‹Ü‚(:€ ·þöÒóÅ‚Þ( disk.nvme0c0n1.out‚19370.72607421875 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.63671875* proc.memory.percent‚17.234110832214355 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚25637.62060546875& disk.nvme0n1.out‚19370.72607421875 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚96.54357433319092 proc.cpu.threads‚17 network.sent‚ 6.272827e+06) disk./.usagePercent‚76.42408357616202% disk./.usageGB‚1173.3007431030273 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875$ disk.nvme1c1n1.out‚ 12038.7890625 disk.rbd1.in‚0" disk.nvme1n1.out‚ 12038.7890625 network.recv‚966385 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125( proc.memory.availableMB‚ 566.30078125 cpu‚25.671764191890073 disk.md0.out‚ 94.44140625 disk.dm-1.out‚ 5568.85546875"Äd˞݂(:“ ·þöÒñ´ÈÞ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚268.378' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.45933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0  šáˆÞ‚(:ý ÆþöÒÆœ“Þ disk.md0.out‚ 95.13671875# disk.dm-0.out‚25638.28466796875! disk.dm-1.out‚5568.9638671875 network.recv‚969575 disk.dm-0.in‚ 8.64453125! proc.memory.rssMB‚ 2823.640625 cpu‚25.670928882611392 proc.cpu.threads‚17 network.sent‚ 6.277691e+06$ disk./.usageGB‚1173.300765991211 disk.nvme0c0n1.in‚7.1875$ disk.nvme1c1n1.out‚ 12039.7265625 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125" disk.nvme1n1.out‚ 12039.7265625$ memory_percent‚96.54359817504883) disk./.usagePercent‚76.42408506700609( disk.nvme0c0n1.out‚19372.03173828125 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 566.296875* proc.memory.percent‚17.234134674072266& disk.nvme0n1.out‚19372.03173828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125L@Cß‚(:’ ÆþöÒ‘‡×Þ gpu.0.gpu‚43.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.50666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Ø¢x2à- _wandbruntime‚3676  _runtime‚3676Ó¥E£Éáà ' lr-AdamW/pg1‚0.00016508666571790699 ' lr-AdamW/pg2‚0.00016508666571790699 trainer/global_step‚3899 % _timestamp‚1.7845287123830266e+09 _runtime‚3676.651948859 _step‚317ÎqÉÎâÈ _runtime‚3676.651948859 _step‚317 ' lr-AdamW/pg1‚0.00016508666571790699 ' lr-AdamW/pg2‚0.00016508666571790699 trainer/global_step‚3899 % _timestamp‚1.7845287123830266e+09½G¯;22ã- _wandbruntime‚3676  _runtime‚3676œsNä'äÞO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129734509858304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38543689 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚318 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚600305576877056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4517990124032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16125834 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70045566 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600305218413568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9320574 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4544157256704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4517990124032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚596817975606988 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4544045696000 _runtime‚ 3676.65213332 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31501877 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚595787245328896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6805257 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31501875 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚596817618006354Vù™.'åãO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚596817618006354 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129734509858304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38543689 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚318 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚600305576877056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4517990124032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16125834 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70045566 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600305218413568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9320574 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4544157256704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4517990124032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚596817975606988 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4544045696000 _runtime‚ 3676.65213332 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31501877 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚595787245328896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6805257 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactiveiç»_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31501875¾ =ðÿ2æ- _wandbruntime‚3677  _runtime‚3677vdãï&çéM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519100222976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129767671516160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519441654272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 trainer/global_step‚3899 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31509668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31509670 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38543202 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚596966514662824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4545162232320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31509670 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16129508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592453901016864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4512273084574 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16129808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70052870 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38553359 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70063029 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚596966157062078 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚595935502791680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600454603014656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70063029 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600454961485312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚595935502791680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134312966011392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4519100222976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4545275883008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38543202 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9322517 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9322814 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134312833748480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129767690128384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Eï½ô&èîM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38543202 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9322517 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9322814 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134312833748480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129767690128384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519100222976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129767671516160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519441654272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 trainer/global_step‚3899 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31509668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31509670 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38543202 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚596966514662824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4545162232320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31509670 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16129508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592453901016864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4512273084574 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16129808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70052870 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38553359 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70063029 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚596966157062078 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚595935502791680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600454603014656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70063029 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600454961485312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚595935502791680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134312966011392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4519100222976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4545275883008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2¿î‘}D2é- _wandbruntime‚3677  _runtime‚36770N(,êÿ 2loss/SRqqt2/assignment_loss‚0.5000525553266091 2loss/FBt1/assignment_loss‚0.016762533017014455 .loss/FRt1/detection_loss‚1.561995285499959 1loss/SRqqt2/detection_loss‚0.9806914082163426 %loss/total_loss‚9.094729852994043 trainer/global_step‚3899 2loss/FBt2/assignment_loss‚0.008476951729464548 /loss/FRt2/detection_loss‚0.5238412772042936 $ _timestamp‚1.784528713014451e+09 _runtime‚3677.282304775 0loss/FRt1/assignment_loss‚0.7646198163474847 1loss/SRqqt1/detection_loss‚0.6318932202472827 /loss/FBt1/detection_loss‚0.5927150601816799 .loss/FBt2/detection_loss‚1.045565277751949 epoch‚9 _step‚320 /loss/FRt2/assignment_loss‚1.936716301895765 2loss/SRqqt1/assignment_loss‚0.53140016557619849•! ë„ epoch‚9 _step‚320 /loss/FRt2/assignment_loss‚1.936716301895765 2loss/SRqqt1/assignment_loss‚0.5314001655761984 2loss/SRqqt2/assignment_loss‚0.5000525553266091 2loss/FBt1/assignment_loss‚0.016762533017014455 .loss/FRt1/detection_loss‚1.561995285499959 1loss/SRqqt2/detection_loss‚0.9806914082163426 %loss/total_loss‚9.094729852994043 trainer/global_step‚3899 2loss/FBt2/assignment_loss‚0.008476951729464548 /loss/FRt2/detection_loss‚0.5238412772042936 $ _timestamp‚1.784528713014451e+09 _runtime‚3677.282304775 0loss/FRt1/assignment_loss‚0.7646198163474847 1loss/SRqqt1/detection_loss‚0.6318932202472827 /loss/FBt1/detection_loss‚0.5927150601816799 .loss/FBt2/detection_loss‚1.045565277751949Àžk¤ÈŒì‚(: ÕþöÒáʪÞ disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚96.54383659362793! proc.memory.rssMB‚ 2823.640625* proc.memory.percent‚17.234134674072266 proc.cpu.threads‚17) disk./.usagePercent‚76.42409003648632% disk./.usageGB‚1173.3008422851562' disk.nvme1c1n1.out‚12040.3662109375' proc.memory.availableMB‚ 566.2578125 cpu‚25.66989720613613 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.out‚ 95.421875 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚25640.09814453125 disk.dm-1.in‚0( disk.nvme0c0n1.out‚19373.90673828125 disk.rbd0.out‚0 network.sent‚ 6.29264e+06 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚12040.3662109375 disk.md0.in‚0! disk.dm-1.out‚5569.0341796875 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚19373.90673828125 network.recv‚972525Þ."îží‚(:“ ÕþöÒËöóÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚289.534' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.51133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eˬî‚(:‚ äþöÒ߃‡Þ disk.md0.in‚0 disk.dm-0.in‚ 8.64453125& disk.nvme0n1.out‚19374.93701171875$ memory_percent‚96.54381275177002 proc.cpu.threads‚17 network.recv‚976476 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 95.8828125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875! proc.memory.rssMB‚ 2823.640625$ disk./.usageGB‚1173.300796508789 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚12041.1181640625 disk.dm-1.out‚5569.189453125( disk.nvme0c0n1.out‚19374.93701171875* proc.memory.percent‚17.234134674072266 cpu‚25.66982103164029 network.sent‚ 6.342241e+06 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0# disk.dm-0.out‚25640.73486328125 disk.nvme0n1.in‚7.1875( proc.memory.availableMB‚ 566.26171875) disk./.usagePercent‚76.42408705479818' disk.nvme1c1n1.out‚12041.1181640625rcï‚(:’ äþöÒç×ÚÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.116' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.038666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â{X‰ð‚(:þ óþöÒ­Ä›Þ disk.dm-1.out‚5569.318359375 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚19376.998046875 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 6.349045e+06 disk.rbd1.in‚0 disk.md0.out‚ 96.12109375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875$ memory_percent‚96.54383659362793' proc.memory.availableMB‚ 566.2578125! proc.memory.rssMB‚ 2823.640625) disk./.usagePercent‚76.42410742966715% disk.nvme1n1.out‚12041.8642578125" disk.dm-0.out‚25642.8896484375' disk.nvme1c1n1.out‚12041.8642578125* proc.memory.percent‚17.234134674072266 proc.cpu.threads‚17 network.recv‚980173 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0& disk.nvme0c0n1.out‚19376.998046875 disk.nvme1c1n1.in‚ 1.45703125 cpu‚25.67094352089641% disk./.usageGB‚1173.3011093139648 disk.rbd1.out‚0 disk.dm-0.in‚ 8.64453125T ñ‚(:’ óþöÒŸíÞ gpu.0.gpu‚73.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.69' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.56333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽB¼2ò- _wandbruntime‚3721  _runtime‚3721% ÿTÈó ' lr-AdamW/pg1‚0.00016470318996712822 ' lr-AdamW/pg2‚0.00016470318996712822 trainer/global_step‚3949 $ _timestamp‚1.784528757175793e+09 _runtime‚3721.443927016 _step‚321t¡¬0ÍôÇ _runtime‚3721.443927016 _step‚321 ' lr-AdamW/pg1‚0.00016470318996712822 ' lr-AdamW/pg2‚0.00016470318996712822 trainer/global_step‚3949 $ _timestamp‚1.784528757175793e+09ÁuÓ(¦2õ- _wandbruntime‚3721  _runtime‚3721›ªUþöÞO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4566988471004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39018198 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4573559225856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16326602 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9434645 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31891525 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4599862311424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607774512622592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607774871086080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607774512622592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603200970436096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607774871086080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31891527 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16326886 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6891957 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603200953396736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135993490942976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39028341 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4573900649984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6891960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31891527 TGDeviceStatsMonit¨) dæor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4599967580672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31891525 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604244992193996 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚599677986683632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚603200953396736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚3721.448593513 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4573559225856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604244634593362 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6891957 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603200953396736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135993490942976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39028341 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4573900649984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6891960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31891527 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4599967580672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31891525 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604244992193996 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚599677986683632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚603200953396736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚3721.448593513 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4573559225856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604244634593362 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4566988471004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39018198 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4573559225856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16326602 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9434645 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31891525 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4599862311424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607774512622592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607774871086080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607774512622592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603200970436096‡;Äî2ø- _wandbruntime‚3722  _runtime‚3722òlÑ ï&ùéM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4574669324800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9436870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6893691 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31899320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _step‚323 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16330561 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603349244938240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70937331 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599825417351104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39027854 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70937331 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70927172 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4574669324800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39027854 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131426685020160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4575010756096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4567756297982 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39038011 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4600975219200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4575010756096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16330842 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136027660239360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70927172 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39038011 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31899318 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚3949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603349227898880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4601082578432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚6043931736490863©Å±í!úîM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4567756297982 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39038011 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4600975219200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4575010756096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16330842 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136027660239360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70927172 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39038011 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31899318 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚3949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603349227898880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4601082578432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚604393173649086 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4574669324800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9436870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6893691 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31899320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _step‚323 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16330561 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603349244938240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70937331 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599825417351104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39027854 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70937331 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70927172 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4574669324800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39027854 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131426685020160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4575010756096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184ÃóL]¤2û- _wandbruntime‚3722  _runtime‚3722~&GJü 0loss/FRt1/assignment_loss‚0.6479403484612704 0loss/FRt2/assignment_loss‚1.8464647543277346 2loss/SRqqt2/assignment_loss‚0.5252591446568387 2loss/FBt2/assignment_loss‚0.023830168847876792 /loss/FRt1/detection_loss‚1.3743695207755082 1loss/SRqqt1/detection_loss‚0.6841074396859999 0loss/SRqqt2/detection_loss‚0.980058488426972 /loss/FBt2/detection_loss‚1.0378098314628006 epoch‚9 2loss/FBt1/assignment_loss‚0.030741179746139845 trainer/global_step‚3949 $ _timestamp‚1.784528757791912e+09 _runtime‚3722.059776314 2loss/SRqqt1/assignment_loss‚0.5538561553052371 /loss/FRt2/detection_loss‚0.5415698438069975 %loss/total_loss‚8.665011764332807 _step‚324 /loss/FBt1/detection_loss‚0.4190048888294318z=Y ý† _step‚324 /loss/FBt1/detection_loss‚0.4190048888294318 0loss/FRt1/assignment_loss‚0.6479403484612704 0loss/FRt2/assignment_loss‚1.8464647543277346 2loss/SRqqt2/assignment_loss‚0.5252591446568387 2loss/FBt2/assignment_loss‚0.023830168847876792 /loss/FRt1/detection_loss‚1.3743695207755082 1loss/SRqqt1/detection_loss‚0.6841074396859999 0loss/SRqqt2/detection_loss‚0.980058488426972 /loss/FBt2/detection_loss‚1.0378098314628006 epoch‚9 2loss/FBt1/assignment_loss‚0.030741179746139845 trainer/global_step‚3949 $ _timestamp‚1.784528757791912e+09 _runtime‚3722.059776314 2loss/SRqqt1/assignment_loss‚0.5538561553052371 /loss/FRt2/detection_loss‚0.5415698438069975 %loss/total_loss‚8.665011764332807Äwa5ߊþ‚(:ÿ ‚ÿöÒÁ¦ÀÞ disk.nvme0n1.in‚7.1875& disk.nvme1c1n1.out‚12042.544921875$ disk.nvme1n1.out‚12042.544921875' proc.memory.availableMB‚ 565.9453125* proc.memory.percent‚17.234158515930176 proc.cpu.threads‚17) disk./.usagePercent‚76.42410742966715 disk.dm-1.out‚ 5569.47265625 disk.md0.out‚ 96.54296875# proc.memory.rssMB‚ 2823.64453125 cpu‚25.670255855562402 network.sent‚ 6.36481e+06 network.recv‚983335 disk.nvme0c0n1.in‚7.1875% disk.nvme0n1.out‚19378.2548828125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚19378.2548828125 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.in‚0" disk.dm-0.out‚25643.7841796875$ memory_percent‚96.54574394226074% disk./.usageGB‚1173.3011093139648‚Ô\ÿ‚(:’ ‚ÿöÒ–€ƒß gpu.0.gpu‚51.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.89' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.29666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eúÇ倂(:‚ ‘ÿöÒ¦ÞŒÞ disk.dm-1.in‚0( disk.nvme0c0n1.out‚19380.91162109375 disk.nvme1c1n1.in‚ 1.45703125# proc.memory.rssMB‚ 2823.64453125 network.sent‚ 6.41629e+06$ disk./.usageGB‚1173.301773071289 disk.md0.out‚ 96.83203125& disk.nvme0n1.out‚19380.91162109375 disk.rbd0.out‚0* proc.memory.percent‚17.234158515930176 proc.cpu.threads‚17 disk.nvme0n1.in‚7.1875 disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875' disk.nvme1c1n1.out‚12043.2119140625 cpu‚25.66945081383724 disk.dm-1.out‚ 5569.54296875 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚12043.2119140625$ memory_percent‚96.54457569122314( proc.memory.availableMB‚ 566.13671875 network.recv‚987877( disk./.usagePercent‚76.4241506641452 disk.md0.in‚0# disk.dm-0.out‚25646.39892578125QIÏaž‚(:“ ‘ÿöÒÒåÕÞ gpu.0.gpu‚52.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.194' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.73133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Éý2‚- _wandbruntime‚3764  _runtime‚3764ê€öˆÈƒÂ _step‚325 ' lr-AdamW/pg1‚0.00016431487700367694 ' lr-AdamW/pg2‚0.00016431487700367694 trainer/global_step‚3999 $ _timestamp‚1.784528800048648e+09 _runtime‚3764.318574683ðÐÍ„Ç ' lr-AdamW/pg1‚0.00016431487700367694 ' lr-AdamW/pg2‚0.00016431487700367694 trainer/global_step‚3999 $ _timestamp‚1.784528800048648e+09 _runtime‚3764.318574683 _step‚325Å\—2…- _wandbruntime‚3764  _runtime‚3764$!Þ®å'†ßO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚615243806831616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611671651180370 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607049537096592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610614695543296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610614695543296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4629469751808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4655690741760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9548916 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32281175 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚610614678503936 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611672008781004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4622471684412 trainer/global_step‚3999 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚615244165295104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16527288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6978660 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133052518254080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39512993 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39502850 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71794170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚615243806831616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137708328683008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4655791816704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16527576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39502850 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9548631 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4629469751808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137708208995840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845288000493636e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4629128327680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6978657 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610614678503936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32281175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39512993 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚326 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32281177 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚326 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32281177 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚615243806831616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611671651180370 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607049537096592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610614695543296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610614695543296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4629469751808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4655690741760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9548916 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32281175 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚610614678503936 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611672008781004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4622471684412 trainer/global_step‚3999 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚615244165295104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16527288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6978660 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133052518254080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39512993 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39502850 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71794170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚615243806831616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137708328683008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4655791816704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16527576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634ÆÞüÛc2ˆ- _wandbruntime‚3764  _runtime‚3764 îpôø‰æM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4630579857920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9551178 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32288968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610762953006080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137742505785856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4623580072776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚615393191432704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4630579857920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133085717136384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71811633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4656910426624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚611820547836840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71801474 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚615393549903360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133085698524160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607196967764064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16531572 _step‚327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71801474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39512506 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610762953006080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4656807261696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16531291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611820190236094 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6980391 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚615393549903360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534ÀS>?ñ&ŠëM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39512506 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610762953006080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4656807261696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16531291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611820190236094 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6980391 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚615393549903360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4630579857920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9551178 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32288968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610762953006080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137742505785856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4623580072776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚615393191432704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4630579857920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133085717136384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71811633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4656910426624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚611820547836840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71801474 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚615393549903360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133085698524160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607196967764064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16531572 _step‚327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71801474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528Ǫ‹B)2‹- _wandbruntime‚3764  _runtime‚3764Ïî{qŒ€ 0loss/FRt1/assignment_loss‚0.8202088054630065 2loss/SRqqt1/assignment_loss‚0.5906627107382519 0loss/SRqqt2/detection_loss‚0.917425626591973 epoch‚9 trainer/global_step‚3999 0loss/FRt2/assignment_loss‚1.8029652274189851 2loss/SRqqt2/assignment_loss‚0.5988753097250608 /loss/FRt1/detection_loss‚1.4381616564745443 /loss/FRt2/detection_loss‚0.5740964186846284 1loss/SRqqt1/detection_loss‚0.5887433889086089 /loss/FBt1/detection_loss‚0.5401970868751409 1loss/FBt2/assignment_loss‚0.07296245418667538 /loss/FBt2/detection_loss‚0.7629188163013294 _runtime‚3764.934219121 1loss/FBt1/assignment_loss‚0.04144119621650193 %loss/total_loss‚8.748658697584707 % _timestamp‚1.7845288006663938e+09 _step‚328Í%´ß … 1loss/FBt2/assignment_loss‚0.07296245418667538 /loss/FBt2/detection_loss‚0.7629188163013294 _runtime‚3764.934219121 1loss/FBt1/assignment_loss‚0.04144119621650193 %loss/total_loss‚8.748658697584707 % _timestamp‚1.7845288006663938e+09 _step‚328 0loss/FRt1/assignment_loss‚0.8202088054630065 2loss/SRqqt1/assignment_loss‚0.5906627107382519 0loss/SRqqt2/detection_loss‚0.917425626591973 epoch‚9 trainer/global_step‚3999 0loss/FRt2/assignment_loss‚1.8029652274189851 2loss/SRqqt2/assignment_loss‚0.5988753097250608 /loss/FRt1/detection_loss‚1.4381616564745443 /loss/FRt2/detection_loss‚0.5740964186846284 1loss/SRqqt1/detection_loss‚0.5887433889086089 /loss/FBt1/detection_loss‚0.5401970868751409Ȥ?!Ž‚(:…  ÿöÒ÷¿®Þ# disk.dm-0.out‚25647.02392578125 disk.dm-1.out‚5569.787109375( disk.nvme0c0n1.out‚19381.82177734375 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚12043.8818359375 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125 disk.nvme0n1.in‚7.1875% disk.nvme1n1.out‚12043.8818359375$ memory_percent‚96.54638767242432# proc.memory.rssMB‚ 2823.64453125* proc.memory.percent‚17.234158515930176 network.sent‚ 6.43205e+06 network.recv‚990936) disk./.usagePercent‚76.42415513667741 disk.rbd0.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 565.83984375 cpu‚25.668510409021597 disk.md0.out‚ 97.1640625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875& disk.nvme0n1.out‚19381.82177734375 disk.rbd0.out‚0 disk.rbd1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1173.3018417358398 disk.md0.in‚0õÄ_•ž‚(:“  ÿöÒ…ÄøÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.681' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.56033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ÿ¡¥‰‚(:þ ¯ÿöÒ¢ùúÝ% disk./.usageGB‚1173.3018035888672 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚25648.94189453125 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚19383.70556640625 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125) disk./.usagePercent‚76.42415265193728 disk.dm-1.out‚5569.861328125 disk.rbd0.out‚0! disk.nvme1n1.out‚ 12044.421875 disk.md0.out‚ 97.36328125$ memory_percent‚97.01511859893799# proc.memory.rssMB‚ 2902.53515625) proc.memory.percent‚17.71566867828369 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚19383.70556640625 disk.rbd1.in‚0( proc.memory.availableMB‚ 489.04296875 cpu‚25.677859922189985 proc.cpu.threads‚17 network.sent‚ 6.483493e+06 network.recv‚994929# disk.nvme1c1n1.out‚ 12044.421875 disk.md0.in‚0kfWZ“‘‚(:ˆ ¯ÿöÒ…±ºÞ gpu.0.gpu‚37.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚119.628' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.876 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tî‰2’- _wandbruntime‚3781  _runtime‚3781‰ÄT9“…{ =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6436790823936462 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948911070823669 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9809358716011047 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2824360430240631 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2199380099773407 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2822820246219635 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3831833004951477 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02550254575908184 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16492736339569092 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21358177065849304 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.729256272315979 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.853579044342041 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.34547093510627747 particle/accuracy_3_of_3‚1 6validation_average_jet_accuracy‚0.3836999237537384 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47231778502464294 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3116685450077057 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09071722626686096 +particle/sensitivity‚0.6334894299507141 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12693054974079132 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6808053255081177 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6540589332580566 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9702855348587036 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.502219557762146 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_2_of_3‚0.5688776969909668 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11699406057596207 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.853579044342041 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9344167113304138 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981386065483093 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9943799376487732 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_3‚1 jet/accuracy_2_of_5‚1 % _timestamp‚1.7845288171322937e+09 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04251754656434059 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6523460149765015 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995432436466217 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6427474617958069 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2712947428226471 *jet/accuracy_1_of_3‚0.9159530401229858 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6920018196105957 +jet/accuracy_3_of_3‚0.07560873031616211 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27237868309020996 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32875964045524597 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03144517540931702 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5920777320861816 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7925313115119934 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6932095289230347 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37830841541290283 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6288814544677734 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7000762224197388 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027278661727905273 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8538171648979187 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6515938639640808 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7899426817893982 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5583333373069763 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6466569900512695 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16249999403953552 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19614189863204956 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 /particle/accuracy_4_of_4‚0.9334194660186768 +particle/specificity‚0.9109169840812683 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043339621275663376 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9831492304801941 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9935734272003174 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6221175193786621 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 )loss/val_total_loss‚8.556175561602345 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27243492007255554 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4937366843223572 _step‚329 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23055827617645264 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5365113615989685 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08227880299091339 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7343745231628418 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22183363139629364 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.634017825126648 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5827606320381165 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7424173355102539 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5864236354827881 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020040331408381462 *jet/accuracy_1_of_2‚0.7838755249977112 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_2_of_2‚0.15886488556861877 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.737015962600708 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7665930390357971 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987590491771698 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6797285676002502 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7719618678092957 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7328087687492371 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10934290289878845 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09736508131027222 (particle/accuracy‚0.8513377904891968 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10007695108652115 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8622071146965027 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4736241400241852 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27243492007255554 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9619917869567871 particle/accuracy_2_of_2‚1 particle/accuracy_5_of_5‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20419831573963165 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4683050811290741 particle/accuracy_1_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025686297565698624 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7112171649932861 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7355146408081055 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18554282188415527 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26420947909355164 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4876401424407959 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7678949236869812 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6414702534675598 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_5_of_5‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 trainer/global_step‚3999 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7541386485099792 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2830790877342224 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 _runtime‚ 3781.4008447 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28928089141845703 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.344489723443985 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987666010856628 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851427674293518 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7390713691711426 *jet/accuracy_2_of_4‚0.7977754473686218 +jet/accuracy_4_of_4‚0.01628045178949833 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8901348114013672 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8608088493347168 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2131737470626831 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6761993765830994 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8824974894523621 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05033835396170616 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22418998181819916 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9138422608375549 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8629063367843628 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 epoch‚9 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6958262920379639 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6436790823936462 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8450492024421692 8"Purity/1l5º“|FRt1SRqqt*FBt/SRqqt_purity‚0.728430986404419 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027010241523385048 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020040331408381462 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6492509245872498 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5361197590827942 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23275111615657806 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8732171654701233 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932630658149719 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21899239718914032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7488089799880981 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3091250956058502 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6787596344947815Ëò=”Š{ +jet/accuracy_3_of_4‚0.09071722626686096 +particle/sensitivity‚0.6334894299507141 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12693054974079132 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6808053255081177 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6540589332580566 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9702855348587036 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.502219557762146 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_2_of_3‚0.5688776969909668 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11699406057596207 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.853579044342041 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9344167113304138 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981386065483093 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9943799376487732 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_3‚1 jet/accuracy_2_of_5‚1 % _timestamp‚1.7845288171322937e+09 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04251754656434059 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6523460149765015 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995432436466217 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6427474617958069 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2712947428226471 *jet/accuracy_1_of_3‚0.9159530401229858 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6920018196105957 +jet/accuracy_3_of_3‚0.07560873031616211 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27237868309020996 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32875964045524597 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03144517540931702 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5920777320861816 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7925313115119934 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6932095289230347 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37830841541290283 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6288814544677734 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7000762224197388 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027278661727905273 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8538171648979187 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6515938639640808 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7899426817893982 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5583333373069763 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6466569900512695 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16249999403953552 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19614189863204956 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 /particle/accuracy_4_of_4‚0.9334194660186768 +particle/specificity‚0.9109169840812683 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043339621275663376 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9831492304801941 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9935734272003174 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6221175193786621 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 )loss/val_total_loss‚8.556175561602345 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27243492007255554 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4937366843223572 _step‚329 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23055827617645264 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5365113615989685 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08227880299091339 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7343745231628418 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22183363139629364 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.634017825126648 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5827606320381165 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7424173355102539 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5864236354827881 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020040331408381462 *jet/accuracy_1_of_2‚0.7838755249977112 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_2_of_2‚0.15886488556861877 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.737015962600708 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7665930390357971 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987590491771698 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6797285676002502 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7719618678092957 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7328087687492371 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10934290289878845 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09736508131027222 (particle/accuracy‚0.8513377904891968 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10007695108652115 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8622071146965027 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4736241400241852 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27243492007255554 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9619917869567871 particle/accuracy_2_of_2‚1 particle/accuracy_5_of_5‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20419831573963165 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4683050811290741 particle/accuracy_1_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025686297565698624 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7112171649932861 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7355146408081055 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18554282188415527 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26420947909355164 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4876401424407959 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7678949236869812 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6414702534675598 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_5_of_5‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 trainer/global_step‚3999 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7541386485099792 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2830790877342224 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 _runtime‚ 3781.4008447 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28928089141845703 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.344489723443985 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987666010856628 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851427674293518 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7390713691711426 *jet/accuracy_2_of_4‚0.7977754473686218 +jet/accuracy_4_of_4‚0.01628045178949833 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8901348114013672 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8608088493347168 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2131737470626831 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6761993765830994 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8824974894523621 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05033835396170616 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22418998181819916 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9138422608375549 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8629063367843628 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 epoch‚9 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6958262920379639 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6436790823936462 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8450492024421692 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.728430986404419 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027010241523385048 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020040331408381462 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6492509245872498 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5361197590827942 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23275111615657806 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8732171654701233 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932630658149719 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21899239718914032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7488089799880981 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3091250956058502 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6787596344947815 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6436790823936462 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948911070823669 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9809358716011047 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2824360430240631 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2199380099773407 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2822820246219635 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3831833004951477 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02550254575908184 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16492736339569092 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21358177065849304 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.729256272315979 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.853579044342041 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.34547093510627747 particle/accuracy_3_of_3‚1 6validation_average_jet_accuracy‚0.3836999237537384 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47231778502464294 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3116685450077057 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1Ép¹òå•‚(:… ¾ÿöÒ¶‡ŸÞ disk.nvme0n1.in‚7.1875' disk.nvme1c1n1.out‚12290.4208984375$ memory_percent‚95.47922611236572* proc.memory.percent‚17.702412605285645 disk.md0.in‚0 disk.md0.out‚ 97.66015625 disk.nvme0c0n1.in‚7.1875 disk.rbd1.out‚0% disk.nvme1n1.out‚12290.4208984375( proc.memory.availableMB‚ 740.68359375 cpu‚25.67709032670433 network.recv‚998308( disk./.usagePercent‚76.4241740207023$ disk./.usageGB‚1173.302131652832 disk.rbd0.in‚0# proc.memory.rssMB‚ 2900.36328125 network.sent‚ 6.486516e+06# disk.dm-0.out‚25649.71923828125& disk.nvme0n1.out‚19384.58447265625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 proc.cpu.threads‚17 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0! disk.dm-1.out‚5815.3134765625( disk.nvme0c0n1.out‚19384.584472656252¼áÇœ–‚(:‘ ¾ÿöÒ÷æß gpu.0.gpu‚11.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.66' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.88666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãî5—‚(:… ÍÿöÒ‰à®Þ( proc.memory.availableMB‚ 740.67578125* proc.memory.percent‚17.702412605285645 proc.cpu.threads‚17) disk./.usagePercent‚76.42418942609103 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚25652.0869140625 disk.nvme0c0n1.in‚7.1875& disk.nvme0c0n1.out‚19386.986328125 cpu‚25.676198368966798 network.sent‚ 6.545528e+06 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚12536.529296875 disk.rbd1.out‚0$ disk.nvme1n1.out‚12536.529296875# proc.memory.rssMB‚ 2900.36328125 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚19386.986328125 network.recv‚ 1.002989e+06 disk.md0.out‚ 97.984375! disk.dm-1.out‚6060.7626953125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.47927379608154% disk./.usageGB‚1173.3023681640625 disk.md0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0Äþ~ ž˜‚(:“ ÍÿöÒÃÃîÞ gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚281.456' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.81866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊßÙ‹™‚(:€ ÜÿöÒÆý’ß network.sent‚ 6.550392e+06$ disk./.usageGB‚1170.585163116455 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚6060.89306640625 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0( disk.nvme1c1n1.out‚12537.34130859375 disk.rbd1.in‚0 network.recv‚ 1.006316e+06( disk./.usagePercent‚76.2472016359847" disk.dm-0.out‚25652.8876953125 disk.nvme0c0n1.in‚7.1875 disk.rbd1.out‚0& disk.nvme1n1.out‚12537.34130859375 disk.md0.in‚0 disk.md0.out‚ 98.48046875" disk.nvme0c0n1.out‚ 19388.15625 disk.rbd0.out‚0& proc.memory.availableMB‚ 740.671875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.nvme0n1.out‚ 19388.15625 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚95.47929763793945# proc.memory.rssMB‚ 2900.36328125* proc.memory.percent‚17.702412605285645 cpu‚25.675564433658206³Æ'ç“š‚(:ˆ ÜÿöÒœê»Þ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚292.407' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚97.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÂÎ2›- _wandbruntime‚3825  _runtime‚3825W=Ȝ ' lr-AdamW/pg2‚0.00016392175176816912 trainer/global_step‚4049 $ _timestamp‚1.784528860799231e+09 _runtime‚3825.067458715 _step‚330 ' lr-AdamW/pg1‚0.00016392175176816912`‚×WÍÇ $ _timestamp‚1.784528860799231e+09 _runtime‚3825.067458715 _step‚330 ' lr-AdamW/pg1‚0.00016392175176816912 ' lr-AdamW/pg2‚0.00016392175176816912 trainer/global_step‚4049ÊY[eÝ2ž- _wandbruntime‚3825  _runtime‚3825ÆÍ×å'ŸßO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40079144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32793635 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845288608014936e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624675762754560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139466448849920 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32793635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139466572731392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616342326478416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4697918185448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4705015267840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134734110420480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32793633 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624676121218048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚621040227624504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40079144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40069001 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚331 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624676121218048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619970764526080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16773192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9703440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134734129032704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624675762754560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72872779 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7069473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40069001 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚619970747486720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚616342309439056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚3825.069566589 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9703716 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4732338429440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚619970764526080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4705015267840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16772913 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7069476 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4698258746722èáHž< äO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚3825.069566589 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9703716 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4732338429440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDW‰ö®#DeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚619970764526080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4705015267840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16772913 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7069476 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4698258746722 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40079144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32793635 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845288608014936e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624675762754560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139466448849920 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32793635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139466572731392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616342326478416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4697918185448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4705015267840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134734110420480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32793633 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624676121218048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚621040227624504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40079144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40069001 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚331 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624676121218048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619970764526080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16773192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9703440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134734129032704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624675762754560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72872779 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7069473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40069001 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚619970747486720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚616342309439056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Ëz•Y2¡- _wandbruntime‚3825  _runtime‚3825mMA&î&¢èM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4706466798080 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9706037 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616489757145888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72890242 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40088814 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72890242 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9705759 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32801426 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40088814 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72880083 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134767290690560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620119039028224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32801428 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620119039028224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7071207 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134767309302784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72880083 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4706125366784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845288614279816e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7071210 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624825505826304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4706125366784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32801426 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚332 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4699026573700 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624825505826304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4699367135086 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40078657 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139500746738176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚620119021988864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16777247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16776966 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896ŒÛêó&£íM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72890242 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9705759 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32801426 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40088814 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72880083 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134767290690560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620119039028224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32801428 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620119039028224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7071207 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134767309302784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72880083 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4706125366784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845288614279816e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7071210 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624825505826304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4706125366784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32801426 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚332 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4699026573700 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624825505826304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4699367135086 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40078657 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139500746738176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚620119021988864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16777247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16776966 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4706466798080 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9706037 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616489757145888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72890242 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40088814 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ìæ òË2¤- _wandbruntime‚3825  _runtime‚3825|ÓÁ ¥„ %loss/total_loss‚9.025078653211704 2loss/SRqqt1/assignment_loss‚0.5965481116211504 2loss/FBt2/assignment_loss‚0.006721367740283313 /loss/FRt1/detection_loss‚1.5580747894105618 1loss/SRqqt1/detection_loss‚0.7094240745227168 1loss/SRqqt2/detection_loss‚0.9521018856076184 /loss/FBt1/detection_loss‚0.5648681364691583 /loss/FBt2/detection_loss‚0.8033985683827528 2loss/SRqqt2/assignment_loss‚0.5823721709626773 trainer/global_step‚4049 _step‚333 epoch‚10 /loss/FRt2/detection_loss‚0.5512780661758763 % _timestamp‚1.7845288614297335e+09 _runtime‚3825.697621394 0loss/FRt1/assignment_loss‚0.8204441420147928 0loss/FRt2/assignment_loss‚1.8506636838817117 2loss/FBt1/assignment_loss‚0.029183656422404614•ò¦‰ %loss/total_loss‚9.025078653211704 2loss/SRqqt1/assignment_loss‚0.5965481116211504 2loss/FBt2/assignment_loss‚0.006721367740283313 /loss/FRt1/detection_loss‚1.5580747894105618 1loss/SRqqt1/detection_loss‚0.7094240745227168 1loss/SRqqt2/detection_loss‚0.9521018856076184 /loss/FBt1/detection_loss‚0.5648681364691583 /loss/FBt2/detection_loss‚0.8033985683827528 2loss/SRqqt2/assignment_loss‚0.5823721709626773 trainer/global_step‚4049 _step‚333 epoch‚10 /loss/FRt2/detection_loss‚0.5512780661758763 % _timestamp‚1.7845288614297335e+09 _runtime‚3825.697621394 0loss/FRt1/assignment_loss‚0.8204441420147928 0loss/FRt2/assignment_loss‚1.8506636838817117 2loss/FBt1/assignment_loss‚0.029183656422404614Íõú2—§‚(:Œ ëÿöÒ¦œÝÞ proc.cpu.threads‚17 network.recv‚ 1.009465e+06( disk.nvme1c1n1.out‚12537.95361328125 disk.md0.in‚0 disk.md0.out‚ 98.79296875( disk.nvme0c0n1.out‚19391.88525390625 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0$ memory_percent‚95.47996520996094 network.sent‚ 6.553314e+06 disk.nvme1c1n1.in‚ 1.45703125 disk.dm-1.in‚0" disk.dm-1.out‚6060.96728515625 disk.nvme0c0n1.in‚7.1875$ proc.memory.availableMB‚740.5625* proc.memory.percent‚17.702412605285645 cpu‚25.67605431989202# disk.dm-0.out‚25656.48486328125) disk./.usagePercent‚76.24720933867907% disk./.usageGB‚1170.5852813720703 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125& disk.nvme0n1.out‚19391.88525390625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2900.36328125 disk.rbd1.out‚0& disk.nvme1n1.out‚12537.95361328125 disk.dm-0.in‚ 8.64453125!‡úŽž¨‚(:“ ëÿöÒ™Í¡Þ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚276.311' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.10366666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©F‘©‚(:† úÿöÒ‹í™Þ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.537' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.179 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÕ/r᪂(:‘ úÿöÒÖš²Þ% disk./.usageGB‚1170.5854606628418( disk.nvme1c1n1.out‚12538.38623046875 cpu‚25.675183136022834 proc.cpu.threads‚17 disk.dm-0.in‚ 8.64453125) disk./.usagePercent‚76.24722101lœäé»695761 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚12538.38623046875 disk.md0.out‚ 99.00390625 disk.nvme0c0n1.in‚7.1875 disk.rbd1.out‚0# disk.dm-0.out‚25657.07470703125" disk.dm-1.out‚6061.10009765625 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚19392.63427734375 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 740.55078125 network.sent‚ 6.617851e+06 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚19392.63427734375$ memory_percent‚95.48003673553467# proc.memory.rssMB‚ 2900.36328125* proc.memory.percent‚17.702412605285645 network.recv‚ 1.013462e+06Ctv«‚(:… ‰€÷ÒÕš¤Þ network.recv‚ 1.017001e+06 disk.md0.in‚0 disk.md0.out‚ 100.20703125 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0& proc.memory.availableMB‚ 740.765625# proc.memory.rssMB‚ 2900.36328125 cpu‚25.674315502108094 network.sent‚ 6.62272e+06% disk./.usageGB‚1168.3698768615723 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚25666.13232421875" disk.nvme0n1.out‚ 19401.6484375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚12540.9912109375 disk.rbd1.in‚0) disk./.usagePercent‚76.10290681397636 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚12540.9912109375 disk.dm-1.in‚0" disk.dm-1.out‚6061.17041015625 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 19401.6484375$ memory_percent‚95.47872543334961* proc.memory.percent‚17.702412605285645 proc.cpu.threads‚17šîˆ£œ¬‚(:‘ ‰€÷Ò•ËðÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.732' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.57733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùN2­- _wandbruntime‚3870  _runtime‚3870 ‹Æ®À trainer/global_step‚4099 $ _timestamp‚1.784528906371622e+09 _runtime‚3870.640040716 _step‚334 & lr-AdamW/pg1‚0.0001635238395103041 & lr-AdamW/pg2‚0.0001635238395103041!‘í˯Štrainer/global_step‚4099 $ _timestamp‚1.784528906371622e+09 _runtime‚3870.640040716 _step‚334 & lr-AdamW/pg1‚0.0001635238395103041 & lr-AdamW/pg2‚0.0001635238395103041Îä1 ¨2°- _wandbruntime‚3870  _runtime‚3870Ç+Ü­ä'±ÞO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16975164 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623713859852016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627384489633280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73747081 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33183285 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9819263 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚628467244211512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9818991 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632145056963584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4760584369664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40553653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40553653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4760584369664 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628467601812146 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4753401398856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627384472593920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632145056963584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141181370369536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚632145415427072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845289063733013e+09 _step‚335 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33183283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275ªyé'²ãO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9818991 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632145056963584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4760584369664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40553653 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40553653 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4760584369664 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628467601812146 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4753401398856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627384472593920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632145056963584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141181370369536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚632145415427072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845289063733013e+09 _step‚335 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33183283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16975164 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623713859852016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627384489633280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73747081 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33183285 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9819263 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚628467244211512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647Ï@Móð2³- _wandbruntime‚3871  _runtime‚3871c„üÅØ%´èM % _timestamp‚1.7845289070323176e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627532764135424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚632294441564672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚628615783267236 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627532747096064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33191076 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73764544 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627532747096064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9821236 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136426304194560 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73754385 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141215668454400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40563309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4789364259840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136426322806784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33191076 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40563309 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚632294800035328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33191078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4762035899904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141215783940096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73764544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9821511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚632294800035328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4762035899904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 trainer/global_step‚4099 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623861273480128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4761694468608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4761694468608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623861290519488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4789461133312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40573466 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16979421 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16979143 _runtime‚3871.300978508 REDeviceStatsMonitor.on_train_batch_end/allocation.âAñlarge_pool.allocated‚33191078 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344Îͦuó&µíM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40563309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4789364259840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136426322806784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33191076 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40563309 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚632294800035328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33191078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4762035899904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141215783940096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73764544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9821511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚632294800035328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4762035899904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 trainer/global_step‚4099 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623861273480128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4761694468608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4761694468608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623861290519488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4789461133312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40573466 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16979421 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16979143 _runtime‚3871.300978508 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33191078 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845289070323176e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627532764135424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚632294441564672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚628615783267236 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627532747096064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33191076 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73764544 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627532747096064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9821236 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136426304194560 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73754385 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141215668454400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528ÐÜÔX/2¶- _wandbruntime‚3871  _runtime‚3871pÁ Ä·ý .loss/FRt2/assignment_loss‚1.87043328745771 2loss/SRqqt2/assignment_loss‚0.6758143415145659 /loss/FRt1/detection_loss‚1.4554757976040398 -loss/FBt1/detection_loss‚0.60277908855119 epoch‚10 2loss/SRqqt1/assignment_loss‚0.6337992634756779 2loss/FBt1/assignment_loss‚0.023083131926151818 2loss/FBt2/assignment_loss‚0.002435159745580291 1loss/SRqqt1/detection_loss‚0.6262687056201544 /loss/FBt2/detection_loss‚0.8681631810172941 _runtime‚3871.305601865 _step‚337 0loss/FRt1/assignment_loss‚0.7960768966330696 /loss/FRt2/detection_loss‚0.5165269872196718 0loss/SRqqt2/detection_loss‚1.025308980642696 #loss/total_loss‚9.0961648214078 trainer/global_step‚4099 % _timestamp‚1.7845289070341957e+09‘=Æ%¸‚ /loss/FRt1/detection_loss‚1.4554757976040398 -loss/FBt1/detection_loss‚0.60277908855119 epoch‚10 2loss/SRqqt1/assignment_loss‚0.6337992634756779 2loss/FBt1/assignment_loss‚0.023083131926151818 2loss/FBt2/assignment_loss‚0.002435159745580291 1loss/SRqqt1/detection_loss‚0.6262687056201544 /loss/FBt2/detection_loss‚0.8681631810172941 _runtime‚3871.305601865 _step‚337 0loss/FRt1/assignment_loss‚0.7960768966330696 /loss/FRt2/detection_loss‚0.5165269872196718 0loss/SRqqt2/detection_loss‚1.025308980642696 #loss/total_loss‚9.0961648214078 trainer/global_step‚4099 % _timestamp‚1.7845289070341957e+09 .loss/FRt2/assignment_loss‚1.87043328745771 2loss/SRqqt2/assignment_loss‚0.6758143415145659ÑC×@d’¹‚(:‡ ˜€÷Òž»Þ disk.md0.out‚ 100.5234375 disk.dm-1.in‚0' proc.memory.availableMB‚ 740.7109375 cpu‚25.673386515152004( disk./.usagePercent‚76.2500190828041# disk./.usageGB‚1170.62841796875 disk.rbd1.in‚0 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚27581.01708984375 disk.dm-1.out‚6061.357421875 disk.nvme0n1.in‚7.1875 proc.cpu.threads‚17 disk.rbd0.out‚0& disk.nvme1c1n1.out‚14454.908203125& disk.nvme0n1.out‚19403.49755859375$ memory_percent‚95.47905921936035* proc.memory.percent‚17.702412605285645 network.sent‚ 6.638485e+06 disk.rbd0.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚19403.49755859375# proc.memory.rssMB‚ 2900.36328125 network.recv‚ 1.020244e+06 disk.nvme1c1n1.in‚ 1.45703125$ disk.nvme1n1.out‚14454.908203125ÃÓÑøžº‚(:“ ˜€÷Ò®š‡ß gpu.0.gpu‚72.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.784' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.26133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 úzJ“»‚(:ˆ §€÷Ò€¥“Þ gpu.0.gpu‚62.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚158.343' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚52.781 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹À„’¼‚(:‡ §€÷Ò̯¨Þ disk.rbd1.out‚0# disk.nvme1n1.out‚14456.30859375' proc.memory.availableMB‚ 740.4609375 cpu‚25.67253294063056 proc.cpu.threads‚17 network.sent‚ 6.690071e+06% disk./.usageGB‚1170.6285667419434 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1c1n1.out‚14456.30859375 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.in‚0 disk.md0.out‚ 100.93359375# disk.dm-0.out‚27587.86767578125 disk.dm-1.in‚0 disk.dm-1.out‚6061.396484375# memory_percent‚95.4805850982666 network.recv‚ 1.024676e+06( disk.nvme0c0n1.out‚19409.88134765625 disk.dm-0.in‚ 8.64453125# proc.memory.rssMB‚ 2900.36328125* proc.memory.percent‚17.702412605285645) disk./.usagePercent‚76.25002877329057& disk.nvme0n1.out‚19409.88134765625ˆ„cª“½‚(:ˆ ¶€÷ÒžÉ€Þ gpu.0.gpu‚90.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.874' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨Å–¾‚(:‹ ¶€÷ÒÝë˜Þ# proc.memory.rssMB‚ 2900.36328125* proc.memory.percent‚17.702412605285645 proc.cpu.threads‚17 network.sent‚ 6.694928e+06$ disk./.usageGB‚1171.585292816162 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚20106.38232421875 disk.rbd0.out‚0$ memory_percent‚95.48063278198242 network.recv‚ 1.02768e+06) disk./.usagePercent‚76.31234605544127 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚28519.22314453125" disk.dm-1.out‚6061.51416015625( disk.nvme0c0n1.out‚20106.38232421875 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.out‚ 101.21875( disk.nvme1c1n1.out‚14691.90771484375 disk.rbd1.in‚0& disk.nvme1n1.out‚14691.90771484375& proc.memory.availableMB‚ 740.453125 cpu‚25.671749177881885 disk.rbd0.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125V JÈ2¿- _wandbruntime‚3916  _runtime‚3916ocWŠÈÀ ' lr-AdamW/pg1‚0.00016312116578724256 ' lr-AdamW/pg2‚0.00016312116578724256 trainer/global_step‚4149 $ _timestamp‚1.784528952091272e+09 _runtime‚3916.359428674 _step‚338©\~9ÍÁÇ ' lr-AdamW/pg1‚0.00016312116578724256 ' lr-AdamW/pg2‚0.00016312116578724256 trainer/global_step‚4149 $ _timestamp‚1.784528952091272e+09 _runtime‚3916.359428674 _step‚338Ò%äÛË2Â- _wandbruntime‚3916  _runtime‚3916 Êã'ÃÝO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33572935 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7242876 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4816494895616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138052156040704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚631085376186256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138052137428480 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚3916.360641726 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33572935 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4816153471488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74621383 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9933095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635894618399154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41038305 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚634798214740480 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 trainer/global_step‚4149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784528952092412e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634798197701120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17176248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7242873 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4844202255360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41038305 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639614709636096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4808884612264 _step‚339 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634798214740480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚634798197701120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41048448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639614351172608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74611238 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33572933 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41048448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142896240706048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9933372 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639614351172608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631085393225616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142896358296064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4844103277568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635894260798520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639614709636096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639614351172608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631085393225616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142896358296064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4844103277568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635894260798520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639614709636096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138052156040704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚631085376186256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138052137428480 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚3916.360641726 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33572935 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4816153471488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74621383 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9933095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_bat˜3v<ÿch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635894618399154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41038305 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚634798214740480 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 trainer/global_step‚4149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784528952092412e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634798197701120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17176248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7242873 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4844202255360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41038305 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639614709636096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4808884612264 _step‚339 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634798214740480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚634798197701120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41048448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145ÓÞ%±Ñ2Å- _wandbruntime‚3916  _runtime‚3916yÚ¥ï&ÆéM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7244610 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845289527230885e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4817263570432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74638846 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚142930537627648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚634946489242624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9935367 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17179974 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636043157454990 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138085336310784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4845323094016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚4149 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚631232823893088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74638846 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74628687 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33580726 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4810333561902 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138085317698560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚340 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33580728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41047961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639763735773696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639764094244352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4845219929088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4809993000516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33580728 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344¨í ô&ÇîM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41058118 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636042799854244 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74638846 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚142930537627648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚634946489242624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9935367 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17179974 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636043157454990 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138085336310784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4845323094016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚4149 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚631232823893088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74638846 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74628687 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33580726 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4810333561902 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138085317698560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚340 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33580728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41047961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639763735773696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639764094244352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4845219929088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4809993000516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33580728 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7244610 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845289527230885e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4817263570432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33962583 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17376922 JBDeviceStatsMonitor.on_train_batch_startO¨ýsX/inactive_split.all.current‚302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7329573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10047047 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41533100 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4872063997440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4871722573312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144611139331072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33962585 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647084003845120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4899988398592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41533100 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75495685 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144611259018240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚643321634986162 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4872063997440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643321277385528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642211922808320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17376620 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚638456909559856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647083645381632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4871722573312 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10047346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33962583 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚638456926599216 $ _timestamp‚1.784528997102902e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚642211939847680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642211939847680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139711169544704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0,ñ é'ÖãO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33962583 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚638456926599216 $ _timestamp‚1.784528997102902e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚642211939847680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642211939847680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139711169544704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚ 3961.3711431 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41522957 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33962583 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17376922 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7329573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10047047 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41533100 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4872063997440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4871722573312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144611139331072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33962585 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647084003845120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4899988398592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41533100 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75495685 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144611259018240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚643321634986162 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4872063997440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643321277385528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642211922808320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17376620 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚638456909559856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647083645381632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4871722573312 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10047346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0מéã2×- _wandbruntime‚3962  _runtime‚3962–Þ/ñ&ØëM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139744349814784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚3962.003471884 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873174103552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7331310 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4865816775310 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75513148 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4872832672256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75502989 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139744331202560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚642360197310464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33970376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10049297 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41542770 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚642360214349824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41532613 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647233029982720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75502989 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17380604 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4872832672256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41542770 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4901210369536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41532613 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4873174103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647233388453376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33970378 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚642360214349824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41532613 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647233029982720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75502989 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17380604 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4872832672256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41542770 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4901210369536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41532613 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4873174103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647233388453376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33970378 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4865816775310 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75513148 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4872832672256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75502989 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180Ø™j¥2Ú- _wandbruntime‚3962  _runtime‚3962q8DÛþ 0loss/FRt1/assignment_loss‚0.7260479051576751 0loss/FBt2/assignment_loss‚0.0659019355519545 /loss/FRt2/detection_loss‚0.4855336629763892 /loss/FBt1/detection_loss‚0.5066873423072846 %loss/total_loss‚8.841277170040735 _step‚345 1loss/FBt1/assignment_loss‚0.03830473047512189 1loss/SRqqt2/detection_loss‚0.8628317337950419 .loss/FBt2/detection_loss‚1.086849772701821 2loss/SRqqt1/assignment_loss‚0.5620831696156263 $ _timestamp‚1.784528997736309e+09 _runtime‚3962.004254355 0loss/FRt2/assignment_loss‚1.8793324107209881 2loss/SRqqt2/assignment_loss‚0.5866489889493518 .loss/FRt1/detection_loss‚1.464959358225424 1loss/SRqqt1/detection_loss‚0.5760961595640569 epoch‚10 trainer/global_step‚4199ºò«p ܃ 0loss/FRt2/assignment_loss‚1.8793324107209881 2loss/SRqqt2/assignment_loss‚0.5866489889493518 .loss/FRt1/detection_loss‚1.464959358225424 1loss/SRqqt1/detection_loss‚0.5760961595640569 epoch‚10 trainer/global_step‚4199 0loss/FRt1/assignment_loss‚0.7260479051576751 0loss/FBt2/assignment_loss‚0.0659019355519545 /loss/FRt2/detection_loss‚0.4855336629763892 /loss/FBt1/detection_loss‚0.5066873423072846 %loss/total_loss‚8.841277170040735 _step‚345 1loss/FBt1/assignment_loss‚0.03830473047512189 1loss/SRqqt2/detection_loss‚0.8628317337950419 .loss/FBt2/detection_loss‚1.086849772701821 2loss/SRqqt1/assignment_loss‚0.5620831696156263 $ _timestamp‚1.784528997736309e+09 _runtime‚3962.004254355Ù78*›Ý‚(: ò€÷Ò¢¯ÀÞ network.recv‚ 1.041503e+06) disk./.usagePercent‚76.42392355889844& disk.nvme0n1.out‚22631.94189453125 disk.rbd0.in‚0( disk.nvme1c1n1.out‚14910.28857421875 disk.rbd1.out‚0 disk.dm-1.in‚0$ memory_percent‚96.98255062103271( disk.nvme0c0n1.out‚22631.94189453125 disk.rbd0.out‚0 disk.rbd1.in‚0& disk.nvme1n1.out‚14910.28857421875 disk.md0.in‚0# proc.memory.rssMB‚ 2900.36328125 cpu‚25.668411515899233% disk./.usageGB‚1173.2982864379883 disk.nvme0n1.in‚7.1875 disk.nvme1n1.in‚ 1.45703125 disk.md0.out‚ 102.765625" disk.dm-1.out‚6065.07763671875* proc.memory.percent‚17.702412605285645 proc.cpu.threads‚17 network.sent‚ 6.783168e+06 disk.nvme0c0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚31256.27783203125( proc.memory.availableMB‚ 494.37890625ÕÞ‚(:’ ò€÷ÒÇŒýÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.005' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.001666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ê=™ß‚(:Ž ÷Ò„ψÞ" disk.dm-1.out‚6065.17138671875& disk.nvme0n1.out‚22633.96728515625' disk.nvme1c1n1.out‚14911.2001953125# proc.memory.rssMB‚ 2900.36328125 cpu‚25.668473231528598 network.recv‚ 1.04591e+06% disk./.usageGB‚1173.2982292175293 disk.md0.in‚0( disk.nvme0c0n1.out‚22633.96728515625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚96.98255062103271( proc.memory.availableMB‚ 494.37890625 disk.rbd1.in‚0% disk.nvme1n1.out‚14911.2001953125" disk.dm-0.out‚31257.9150390625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0 disk.md0.out‚ 103.33203125 disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875* proc.memory.percent‚17.702412605285645 proc.cpu.threads‚17 network.sent‚ 6.833529e+06) disk./.usagePercent‚76.42391983178827 disk.nvme1n1.in‚ 1.45703125%'œà‚(:‘ ÷ÒªöÇÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚286.133' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.37766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{¼Žá‚(:ƒ ÷Ò“àãÝ# proc.memory.rssMB‚ 2900.36328125 proc.cpu.threads‚17) disk./.usagePercent‚76.42393821886513 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.in‚0 disk.dm-1.out‚6065.412109375 cpu‚25.669092047958163 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚14912.787109375 disk.rbd1.out‚0 disk.md0.out‚ 104.56640625' disk.nvme0c0n1.out‚22635.9150390625* proc.memory.percent‚17.702412605285645 network.sent‚ 6.836594e+06 network.recv‚ 1.049419e+06$ disk./.usageGB‚1173.298511505127 disk.nvme1n1.in‚ 1.45703125$ disk.nvme1n1.out‚14912.787109375 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚31258.6806640625 disk.nvme0c0n1.in‚7.1875% disk.nvme0n1.out‚22635.9150390625 disk.rbd1.in‚0 disk.dm-1.in‚0$ memory_percent‚96.98257446289062# proc.memory.availableMB‚494.375Éê ¶“â‚(:ˆ ÷Ò£‰¶Þ gpu.0.gpu‚36.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚162.456' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚54.152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úøp2ã- _wandbruntime‚4007  _runtime‚4007¯áo<Éäà ' lr-AdamW/pg1‚0.00016230163770161203 ' lr-AdamW/pg2‚0.00016230163770161203 trainer/global_step‚4249 % _timestamp‚1.7845290429741209e+09 _runtime‚4007.242299385 _step‚346µ«ÑðÎåÈ ' lr-AdamW/pg1‚0.00016230163770161203 ' lr-AdamW/pg2‚0.00016230163770161203 trainer/global_step‚4249 % _timestamp‚1.7845290429741209e+09 _runtime‚4007.242299385 _step‚346ÚFŒS¯2æ- _wandbruntime‚4007  _runtime‚4007,Oçæ'çàO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10161220 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7416276 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649625664954880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649625647915520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4927633099264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7416273 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _runtime‚4007.243469147 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654552939590656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42017752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645828459972816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34352233 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76369987 _step‚347 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34352235 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42007609 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955941737984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645828442933456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76359842 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42017752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚654552939590656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146326124786688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654553298054144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34352235 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42007609 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955941737984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645828442933456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76359842 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42017752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚654552939590656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146326124786688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654553298054144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7416276 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649625664954880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649625647915520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4927633099264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7416273 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _runtime‚4007.243469147 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654552939590656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42017752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645828459972816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34352233 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76369987 _step‚347 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76387450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 11sk…—¶#7582848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141403363318784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4920959427332 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚654702324191744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76377291 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34360028 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10163537 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649773922417664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚348 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42017265 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17581547 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4921299988718 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7418010 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚650896833028260 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928401774080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚650897190629006 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4928743205376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146360302053888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654702682662400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34360026 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956957347328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645975873600928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42027422 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76387450 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7418007 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146360419636736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141403344706560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34360026 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17581255 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚4249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚654702682662400 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚348 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42017265 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17581547 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4921299988718 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7418010 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚650896833028260 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928401774080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚650897190629006 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4928743205376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146360302053888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654702682662400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34360026 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956957347328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645975873600928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42027422 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76387450 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7418007 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146360419636736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141403344706560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34360026 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17581255 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚4249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚654702682662400 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76387450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141403363318784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4920959427332 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚654702324191744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76377291 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34360028 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10163537 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649773922417664 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148040846784512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚653199993346416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34741885 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143029177940480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚653199976307056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4983202201088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657039390062080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17778749 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7502976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4975674813762 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4982860776960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5011668844032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42492261 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662022233799680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34741883 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10275485 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982860776960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42502404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143029196552704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 4053.67963811 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148040979054592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657039373022720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658175310559544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34741885 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segme¸Y˜Ý nt.large_pool.peak‚1180 % _timestamp‚1.7845290894092717e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657039373022720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7502973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662022233799680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34741883 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10275485 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982860776960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42502404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143029196552704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 4053.67963811 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148040979054592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657039373022720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658175310559544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34741885 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845290894092717e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657039373022720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7502973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148040846784512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚653199993346416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34741885 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143029177940480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚653199976307056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4983202201088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634ß©ËT2û- _wandbruntime‚4054  _runtime‚4054Ðhjð&üêM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚658324207216014 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17782794 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662171976871424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657187664564224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4984312307200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 138554368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148075137021440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5012778810880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚4054.314954217 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77251593 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42512074 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42501917 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42512074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143062376822784 $ _timestamp‚1.784529090046602e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10278084 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42501917 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34749678 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4976442640740 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148075275575808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657187647524864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7504707 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5012898753024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4976783202126 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34749676 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77261752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4984312307200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17782501 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10277794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚653347424013888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657187664564224 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34749676 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4299 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77251593 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658323849615268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662171976871424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77261752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34749678 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34749676 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4299 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77251593 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658323849615268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662171976871424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77261752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34749678 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77251593 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42512074 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42501917 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42512074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143062376822784 $ _timestamp‚1.784529090046602e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10278084 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42501917 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34749678 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4976442640740 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148075275575808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657187647524864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7504707 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5012898753024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4976783202126 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34749676 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77261752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4984312307200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17782501 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10277794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚653347424013888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.alloca+V³‘ted‚657187664564224 F2Š- _wandbruntime‚4099  _runtime‚4099”oç'‹áO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚669491886472192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78108446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664453115169280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35131535 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144688210056704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚664453098129920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845291354934278e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚664453115169280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚4349 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚665602684747186 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5038771302912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35131533 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7589676 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10389944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚669491528008704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚660571509680656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42976913 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17979321 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5030817465896 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669491528008704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4099.761572332 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5038771302912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5067616235008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚665602327146552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17979620 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42976913 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10389648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149755700313088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚665602327146552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17979620 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42976913 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10389648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149755700313088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35131535 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144688210056704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚664453098129920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845291354934278e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚664453115169280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚4349 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚665602684747186 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5038771302912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35131533 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7589676 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10389944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚669491528008704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚660571509680656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42976913 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17979321 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5030817465896 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669491528008704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4099.761572332 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5038771302912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5067616235008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452㻂4õ2- _wandbruntime‚4100  _runtime‚4100ãzîU«ŽëM I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78136054 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42986569 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35139328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚664601372632064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5068624733696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4100.394550766 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669641271080448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42996726 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78125895 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144721390326784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5039881409024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5039881409024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17983314 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7591410 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚664601389671424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5068729995776 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚665750866202276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5039539977728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42986569 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78136054 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚665751223803022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35139328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5031925854148 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35139326 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚660718957387488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10391907 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149790120322560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144721371714560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669641271080448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42996726 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78125895 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144721390326784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5039881409024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5039881409024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17983314 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7591410 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚664601389671424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5068729995776 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚665750866202276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5039539977728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42986569 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78136054 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚665751223803022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35139328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5031925854148 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35139326 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚660718957387488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10391907 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149790120322560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144721371714560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35521183 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671866823237120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5086641240578 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚4146.042902648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78982748 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5093998980608 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43471708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚676961180681216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5123373954048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7676376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10504460 _step‚359 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5094340404736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151470578902528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146347223560704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚676960822217728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35521185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671866823237120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667943060093616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5093998980608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845291817746613e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43461565 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146347204948480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151470694395392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚673029343733560 trainer/global_step‚4399 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7676373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚676960822217728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5086300679304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67696082221772kBú;8 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5086300679304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35521183 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671866823237120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5086641240578 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚4146.042902648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78982748 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5093998980608 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43471708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚676961180681216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5123373954048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7676376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10504460 _step‚359 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5094340404736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151470578902528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146347223560704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚676960822217728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35521185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671866823237120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667943060093616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0çþÿŸ³2Ÿ- _wandbruntime‚4146  _runtime‚4146„) ñ& ëM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672015097739264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚668090490761088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35528978 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5095109079552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4146.677142802 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43471221 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚151504868467712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672015114778624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10506725 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5095450510848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673177882789284 % _timestamp‚1.7845291824086115e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7678107 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35528978 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5095450510848 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18184549 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚4399 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79010356 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35528976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79010356 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146380403830784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79000197 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚677110206818816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672015097739264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5095109079552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146380385218560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43481378 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43471221 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7678110 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5087749628942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10506442 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5124594802688 _step‚360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43481378 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚677110565289472 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151504998633472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5087409067556 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5124483249152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79000197 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18184835 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672015114778624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4146.677142802 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43471221 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚151504868467712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672015114778624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10506725 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5095450510848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673177882789284 % _timestamp‚1.7845291824086115e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7678107 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35528978 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5095450510848 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18184549 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚4399 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79010356 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35528976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79010356 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146380403830784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79000197 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚677110206818816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672015097739264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5095109079552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146380385218560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43481378 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43471221 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7678110 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5087749628942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10506442 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5124594802688 _step‚360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43481378 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚677110565289472 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151504998633472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5087409067556 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5124483249152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79000197 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18184835 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672015114778624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272èºib¿2¢- _wandbruntime‚4146  _runtime‚4146°§m•£ -loss/FRt1/detection_loss‚1.44143362296841 % _timestamp‚1.7845291824101841e+09 0loss/FRt2/assignment_loss‚1.8705583974307245 2loss/SRqqt1/assignment_loss‚0.5685428146278922 /loss/FRt2/detection_loss‚0.5408095785263414 /loss/FBt2/detection_loss‚1.0661117051994162 trainer/global_step‚4399 _step‚361 /loss/FBt1/detection_loss‚0.5011594130733094 1loss/SRqqt1/detection_loss‚0.5845190749729137 1loss/SRqqt2/detection_loss‚0.9480136384012973 %loss/total_loss‚8.934332890611117 epoch‚10 _runtime‚4146.678200014 0loss/FRt1/assignment_loss‚0.8100286302743135 2loss/SRqqt2/assignment_loss‚0.5218084283118584 1loss/FBt1/assignment_loss‚0.07770870242614898 2loss/FBt2/assignment_loss‚0.003638884398492946Êþ´L ¤† epoch‚10 _runtime‚4146.678200014 0loss/FRt1/assignment_loss‚0.8100286302743135 2loss/SRqqt2/assignment_loss‚0.5218084283118584 1loss/FBt1/assignment_loss‚0.07770870242614898 2loss/FBt2/assignment_loss‚0.003638884398492946 -loss/FRt1/detection_loss‚1.44143362296841 % _timestamp‚1.7845291824101841e+09 0loss/FRt2/assignment_loss‚1.8705583974307245 2loss/SRqqt1/assignment_loss‚0.5685428146278922 /loss/FRt2/detection_loss‚0.5408095785263414 /loss/FBt2/detection_loss‚1.0661117051994162 trainer/global_step‚4399 _step‚361 /loss/FBt1/detection_loss‚0.5011594130733094 1loss/SRqqt1/detection_loss‚0.5845190749729137 1loss/SRqqt2/detection_loss‚0.9480136384012973 %loss/total_loss‚8.934332890611117éð f•¥‚(:Š ¦‚÷Ò✄Þ( disk.nvme0c0n1.out‚22650.36376953125 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.71852970123291 network.sent‚ 7.0593e+06 network.recv‚ 1.085775e+06 disk.rbd1.out‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875$ memory_percent‚97.00047969818115 cpu‚25.686861402251296 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.dm-0.out‚31272.30908203125! disk.dm-1.out‚6066.2705078125& disk.nvme0n1.out‚22650.36376953125( proc.memory.availableMB‚ 491.44140625# proc.memory.rssMB‚ 2903.00390625 disk.md0.out‚ 107.19140625 disk.dm-0.in‚ 8.64453125) disk./.usagePercent‚76.42405425622863# disk./.usageGB‚1173.30029296875' disk.nvme1c1n1.out‚14918.6064453125 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1n1.out‚14918.6064453125ÀÖ ³ž¦‚(:“ ¦‚÷Ò¶õÞ gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚281.699' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.89966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I·Ûí2§- _wandbruntime‚4163  _runtime‚4163óI?¯µ¨úz ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.609375 % _timestamp‚1.7845291997089229e+09 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2710686922073364 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2178960144519806 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6974247694015503 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5851176381111145 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5984817147254944 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7900345325469971 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7072194218635559 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14238111674785614 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29009005427360535 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.75065016746521 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0298962090164423 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7749520540237427 ,jetH¬ààË6/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3357391655445099 (particle/accuracy‚0.8537405133247375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029736774042248726 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6902175545692444 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 trainer/global_step‚4399 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2903468906879425 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9543935060501099 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3546159267425537 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7464797496795654 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6116982698440552 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5042545199394226 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.09288172423839569 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7124722003936768 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7590659856796265 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9752362370491028 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7541518211364746 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48842594027519226 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988015294075012 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5535696744918823 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6229121685028076 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2256786972284317 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3165682554244995 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2280794382095337 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963803887367249 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20816396176815033 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2909236252307892 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02658792957663536 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6578116416931152 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11922584474086761 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7491904497146606 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7421936988830566 .particle/accuracy_4_of_4‚0.945543646812439 _runtime‚4163.977495102 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19965708255767822 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6638180613517761 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_4_of_5‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46766185760498047 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6972087025642395 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4811035692691803 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6758847236633301 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 particle/accuracy_2_of_4‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.618200957775116 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.615014374256134 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _step‚362 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917343854904175 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8788359761238098 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04938431456685066 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5615561604499817 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968928098678589 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08436991274356842 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18331262469291687 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24287216365337372 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29024264216423035 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2922292947769165 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7001945376396179 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9105774164199829 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6889011263847351 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23168013989925385 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2795080542564392 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2795080542564392 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6628598570823669 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16171380877494812 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.043732382357120514 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982002973556519 *jet/accuracy_2_of_2‚0.1559443175792694 'particle/f_score‚0.6547778248786926 (loss/val_total_loss‚8.51750409244772 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6569599509239197 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8172335624694824 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.669435441493988 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6712962985038757 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7909104228019714 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9292244911193848 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38724082708358765 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_4_of_5‚1 epoch‚10 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963217973709106 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2152186781167984 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8914911150932312 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8922902345657349 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8935940265655518 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8935940265655518 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1350979506969452 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8692681193351746 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5796127319335938 +jet/accuracy_3_of_3‚0.07453590631484985 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2397220879793167 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27585160732269287 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9911690950393677 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3898390233516693 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7690526247024536 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35107067227363586 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955404996871948 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037934642285108566 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22152893245220184 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 .particle/accuracy_3_of_4‚0.997619092464447 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2792325019836426 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7020808458328247 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8500707745552063 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02951672486960888 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7549718618392944 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8500707745552063 +particle/sensitivity‚0.6459078192710876 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7953238487243652 particle/accuracy_2_of_3‚1 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.296875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12564653158187866 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02959412708878517 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09540928900241852 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.887679398059845 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969092011451721 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6465057730674744 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5358333587646484 particle/accuracy_1_of_1‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4901120662689209 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11749999225139618 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6712962985038757 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8135531544685364 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45659661293029785 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6324814558029175 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45983758568763733 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.749312698841095 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6628598570823669 7validation_average_jet_accuracy‚0.39341360330581665 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5474095344543457 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8962163329124451 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02681943029165268 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7430257797241211 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02681943029165268qÈè…=©ÿz 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.669435441493988 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6712962985038757 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7909104228019714 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9292244911193848 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38724082708358765 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_4_of_5‚1 epoch‚10 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963217973709106 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2152186781167984 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8914911150932312 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8922902345657349 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8935940265655518 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8935940265655518 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1350979506969452 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8692681193351746 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5796127319335938 +jet/accuracy_3_of_3‚0.07453590631484985 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2397220879793167 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27585160732269287 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9911690950393677 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3898390233516693 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7690526247024536 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35107067227363586 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955404996871948 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037934642285108566 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22152893245220184 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 .particle/accuracy_3_of_4‚0.997619092464447 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2792325019836426 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7020808458328247 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8500707745552063 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02951672486960888 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7549718618392944 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8500707745552063 +particle/sensitivity‚0.6459078192710876 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7953238487243652 particle/accuracy_2_of_3‚1 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.296875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12564653158187866 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02959412708878517 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09540928900241852 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.887679398059845 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969092011451721 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6465057730674744 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5358333587646484 particle/accuracy_1_of_1‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4901120662689209 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11749999225139618 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6712962985038757 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8135531544685364 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45659661293029785 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6324814558029175 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45983758568763733 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.749312698841095 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6628598570823669 7validation_average_jet_accuracy‚0.39341360330581665 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5474095344543457 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8962163329124451 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02681943029165268 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7430257797241211 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02681943029165268 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.609375 % _timestamp‚1.7845291997089229e+09 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2710686922073364 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2178960144519806 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6974247694015503 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5851176381111145 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5984817147254944 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7900345325469971 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7072194218635559 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14238111674785614 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29009005427360535 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.75065016746521 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0298962090164423 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7749520540237427 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3357391655445099 (particle/accuracy‚0.8537405133247375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029736774042248726 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6902175545692444 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 trainer/global_step‚4399 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2903468906879425 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9543935060501099 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3546159267425537 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7464797496795654 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6116982698440552 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5042545199394226 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.09288172423839569 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7124722003936768 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7590659856796265 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9752362370491028 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7541518211364746 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48842594027519226 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988015294075012 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5535696744918823 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6229121685028076 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2256786972284317 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3165682554244995 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2280794382095337 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963803887367249 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20816396176815033 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2909236252307892 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02658792957663536 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6578116416931152 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11922584474086761 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7491904497146606 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7421936988830566 .particle/accuracy_4_of_4‚0.945543646812439 _runtime‚4163.977495102 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19965708255767822 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6638180613517761 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_4_of_5‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46766185760498047 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6972087025642395 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4811035692691803 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6758847236633301 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 particle/accuracy_2_of_4‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.618200957775116 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.615014374256134 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _step‚362 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917343854904175 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8788359761238098 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04938431456685066 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5615561604499817 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968928098678589 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08436991274356842 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18331262469291687 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24287216365337372 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29024264216423035 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2922292947769165 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7001945376396179 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9105774164199829 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6889011263847351 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23168013989925385 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2795080542564392 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2795080542564392 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6628598570823669 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16171380877494812 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.043732382357120514 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982002973556519 *jet/accuracy_2_of_2‚0.1559443175792694 'particle/f_score‚0.6547778248786926 (loss/val_total_loss‚8.51750409244772 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6569599509239197 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8172335624694824êNELŒª‚(: µ‚÷Òî’Þ disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚15164.5712890625 disk.rbd1.in‚0 cpu‚25.690848522019923% disk./.usageGB‚1173.3001976013184 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚ 31274.4453125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚22652.47265625 disk.nvme0n1.in‚7.1875# disk.nvme0n1.out‚22652.47265625$ memory_percent‚95.48192024230957 network.sent‚ 7.121651e+06 network.recv‚ 1.090097e+06 disk.md0.out‚ 107.43359375! disk.dm-1.out‚6311.6767578125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚15164.5712890625' proc.memory.availableMB‚ 740.2421875 disk.nvme1c1n1.in‚ 1.45703125! proc.memory.rssMB‚ 2903.015625) proc.memory.percent‚17.71860122680664 proc.cpu.threads‚17) disk./.usagePercent‚76.42404804437834 disk.md0.in‚0…u»äž«‚(:“ µ‚÷Ò„Ô‰ß gpu.0.gpu‚91.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚279.553' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.18433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡} ÿ¬‚(:… Ä‚÷Òè‡Þ) disk./.usagePercent‚76.42404829285235 disk.md0.in‚0 disk.md0.out‚ 107.89453125 disk.dm-1.out‚ 6375.84375 disk.nvme0c0n1.in‚7.1875 disk.nvme1n1.in‚ 1.45703125! proc.memory.rssMB‚ 2903.015625 network.sent‚ 7.178608e+06& disk.nvme0n1.out‚22653.89794921875 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚15229.330078125) proc.memory.percent‚17.71860122680664 proc.cpu.threads‚17% disk./.usageGB‚1173.3002014160156 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0$ disk.nvme1n1.out‚15229.330078125 network.recv‚ 1.093829e+06# disk.dm-0.out‚31275.45849609375( disk.nvme0c0n1.out‚22653.89794921875 disk.nvme0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚95.48194408416748( proc.memory.availableMB‚ 740.23828125 cpu‚25.690047012212283ÀN"©ž­‚(:“ Ä‚÷ÒÁŒõÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚119.498' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.83266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú*ó¢'®‚(:‰ Ó‚÷Ò³ÄÞ disk.md0.out‚ 108.33984375 network.recv‚ 1.097302e+06 disk.nvme0n1.in‚7.1875 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125) proc.memory.percent‚17.71860122680664 cpu‚25.689133764826746 proc.cpu.threads‚17 network.sent‚ 7.18#qÏAm4281e+06) disk./.usagePercent‚76.42409103038237 disk.rbd0.out‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚15411.6103515625$ memory_percent‚95.48194408416748( proc.memory.availableMB‚ 740.23828125! proc.memory.rssMB‚ 2903.015625% disk./.usageGB‚1173.3008575439453 disk.rbd0.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚31278.54541015625 disk.dm-1.in‚0 disk.dm-1.out‚ 6557.2265625 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚22657.04638671875& disk.nvme0n1.out‚22657.04638671875 disk.nvme1c1n1.in‚ 1.45703125' disk.nvme1c1n1.out‚15411.6103515625 disk.md0.in‚0XòmÜž¯‚(:“ Ó‚÷Ò•ÑïÞ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚281.684' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.89466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09µ6D2°- _wandbruntime‚4210  _runtime‚4210-utDZÁ _step‚363 & lr-AdamW/pg1‚0.0001606066022031318 & lr-AdamW/pg2‚0.0001606066022031318 trainer/global_step‚4449 % _timestamp‚1.7845292459644663e+09 _runtime‚4210.233557496 j–óÌ²Æ & lr-AdamW/pg1‚0.0001606066022031318 & lr-AdamW/pg2‚0.0001606066022031318 trainer/global_step‚4449 % _timestamp‚1.7845292459644663e+09 _runtime‚4210.233557496 _step‚363ëM¯2³- _wandbruntime‚4210  _runtime‚4210¶Á£tæ'´àO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5199939301376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681222892219904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681222909259264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5170227344896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44037859 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686392778140672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153228854006272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18422800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7767189 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚677235832436080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686393136604160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚677235849475440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44027716 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚682398277778328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80061357 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5169885920768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5200038279168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4449 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681222909259264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5169885920768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10655339 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80071502 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚148028797114880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4210.233618836 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686393136604160 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686392778140672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36033643ŒÛú–ë'µåO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36033643 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5199939301376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681222892219904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681222909259264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5170227344896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44037859 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686392778140672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153228854006272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18422800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7767189 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚677235832436080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686393136604160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚677235849475440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44027716 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚682398277778328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80061357 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5169885920768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5200038279168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4449 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681222909259264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5169885920768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10655339 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80071502 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚148028797114880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4210.233618836 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686393136604160 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686392778140672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452ìÑ[Ñ2¶- _wandbruntime‚4210  _runtime‚4210òŠð&·êM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44047529 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10657878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7768923 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚682546816834164 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚365 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44037372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80088965 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18426535 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80078806 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681371183761408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686542162741760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5201147784192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚681371166722048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚677383263103552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80088965 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36041436 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148061977384960 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4210.862676824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528Sñ£&¾¸ïM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7768926 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5163536691252 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36041434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111291392 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_aù•97$end/segment.small_pool.peak‚272 trainer/global_step‚4449 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80088965 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36041436 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148061977384960 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4210.862676824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44047529 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10657878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7768923 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚682546816834164 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚365 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44037372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80088965 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18426535 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80078806 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681371183761408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686542162741760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5201147784192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚681371166722048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚677383263103552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36423293 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693862072349696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255903720960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255794257408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5217570955022 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚693862430813184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5217911516296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845292916051836e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44522511 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚693862430813184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36423293 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36423291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154943732952064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚684607365809680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36423291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚684607382849040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5225796446720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5225455022592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18623968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚689825294365336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272¥QŸ§ÇæO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36423291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154943732952064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚684607365809680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36423291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚684607382849040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5225796446720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5225455022592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18623968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚689825294365336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80935659 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7853889 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10770079 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7853892 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10770375 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44522511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149687829231104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36423293 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693862072349696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255903720960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255794257408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5217570955022 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚693862430813184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5217911516296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845292916051836e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44522511 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚693862430813184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36423293 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ð‰8Ùƒ2È- _wandbruntime‚4256  _runtime‚4256~=)ñ&ÉëM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5226565121536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688784908868608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 % _timestamp‚1.7845292922380297e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚688784891829248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80953108 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚694011456950784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18627942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154977901146624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44522024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44532181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚684754796477152 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80963267 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36431086 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694011815421440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80963267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5218679343274 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚688784891829248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44522024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149720990888960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44532181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36431086 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36431084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10772603 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18628229 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154978029215232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚684754813516512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5226906552832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149721009501184 trainer/global_step‚4499 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚694011815421440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80953108 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚689973475820426 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5226906552832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5226565121536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688784908868608 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80953108 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚694011456950784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18627942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154977901146624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44522024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44532181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚684754796477152 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80963267 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36431086 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694011815421440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80963267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5218679343274 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚688784891829248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44522024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149720990888960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44532181 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36431086 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36431084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10772603 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18628229 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154978029215232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚684754813516512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5226906552832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149721009501184 trainer/global_step‚4499 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚694011815421440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80953108 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚689973475820426ñýO@É2Ë- _wandbruntime‚4256  _runtime‚4256ãvÓÌ 0loss/FRt2/assignment_loss‚1.8189434322541607 1loss/SRqqt1/assignment_loss‚0.548230488097493 1loss/FBt1/assignment_loss‚0.07346387598566193 1loss/SRqqt1/detection_loss‚0.6423734902973692 epoch‚11 trainer/global_step‚4499 2loss/SRqqt2/assignment_loss‚0.5363113634915256 /loss/FRt1/detection_loss‚1.4299177350416556 /loss/FRt2/detection_loss‚0.5387212082993914 /loss/FBt1/detection_loss‚0.5028501255084958 .loss/FBt2/detection_loss‚1.054053174976546 1loss/SRqqt2/detection_loss‚0.9622391350535003 %loss/total_loss‚8.846484348032089 % _timestamp‚1.7845292922401006e+09 0loss/FRt1/assignment_loss‚0.7154245345954977 2loss/FBt2/assignment_loss‚0.023955784430791792 _runtime‚4256.508250361 _step‚370× Ž‹ ͆ 0loss/FRt1/assignment_loss‚0.7154245345954977 2loss/FBt2/assignment_loss‚0.023955784430791792 _runtime‚4256.508250361 _step‚370 0loss/FRt2/assignment_loss‚1.8189434322541607 1loss/SRqqt1/assignment_loss‚0.548230488097493 1loss/FBt1/assignment_loss‚0.07346387598566193 1loss/SRqqt1/detection_loss‚0.6423734902973692 epoch‚11 trainer/global_step‚4499 2loss/SRqqt2/assignment_loss‚0.5363113634915256 /loss/FRt1/detection_loss‚1.4299177350416556 /loss/FRt2/detection_loss‚0.5387212082993914 /loss/FBt1/detection_loss‚0.5028501255084958 .loss/FBt2/detection_loss‚1.054053174976546 1loss/SRqqt2/detection_loss‚0.9622391350535003 %loss/total_loss‚8.846484348032089 % _timestamp‚1.7845292922401006e+09ò¤ú*0΂(:‚ ƒ÷Òþð½Þ network.recv‚ 1.110529e+06) disk./.usagePercent‚76.16003894044957" disk.dm-1.out‚6557.93994140625 disk.nvme0c0n1.in‚7.1875 disk.nvme1c1n1.in‚ 1.45703125 network.sent‚ 7.271342e+06$ disk./.usageGB‚1169.246997833252 disk.md0.in‚0 disk.md0.out‚ 112.90625$ disk.nvme0n1.out‚22672.310546875 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0& disk.nvme1n1.out‚15419.52001953125$ memory_percent‚96.98805809020996! proc.memory.rssMB‚ 2903.015625 proc.cpu.threads‚17 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚31291.62890625& disk.nvme0c0n1.out‚22672.310546875( disk.nvme1c1n1.out‚15419.52001953125 disk.nvme1n1.in‚ 1.45703125' proc.memory.availableMB‚ 493.4765625) proc.memory.percent‚17.71860122680664 cpu‚25.68583376823441Kä>ߛς(: ƒ÷ÒÐúŠß gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.65' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.54999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹‘uŽÐ‚(:ƒ žƒ÷ÒÈ´›Þ$ disk.nvme1n1.out‚15420.283203125 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875% disk.nvme0n1.out‚22673.3583984375( proc.memory.availableMB‚ 493.42578125 network.recv‚ 1.114961e+06) disk./.usagePercent‚76.16004788551399% disk./.usageGB‚1169.2471351623535 disk.rbd0.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚31292.23046875' disk.nvme0c0n1.out‚22673.3583984375 disk.nvme1c1n1.in‚ 1.45703125 disk.md0.in‚0! disk.dm-1.out‚6558.1474609375# memory_percent‚96.9883680343628! proc.memory.rssMB‚ 2903.015625 cpu‚25.685012810165393 proc.cpu.threads‚17 network.sent‚ 7.320879e+06 disk.rbd1.out‚0 disk.md0.out‚ 113.3828125 disk.nvme0c0n1.in‚7.1875) proc.memory.percent‚17.71860122680664 disk.rbd0.in‚0& disk.nvme1c1n1.out‚15420.283203125 disk.rbd1.in‚0m»L_œÑ‚(:‘ žƒ÷Òø°ôÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚122.446' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.81533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌX*‡Ò‚(:ü ­ƒ÷ÒኦÞ$ disk.nvme0c0n1.out‚ 22678.1640625 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125! proc.memory.rssMB‚ 2903.015625) proc.memory.percent‚17.71860122680664 network.sent‚ 7.327682e+06 network.recv‚ 1.118368e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875" disk.nvme0n1.out‚ 22678.1640625 disk.rbd0.out‚0 disk.rbd1.in‚0# memory_percent‚96.9883918762207 disk.md0.out‚ 113.73046875 disk.dm-0.in‚ 8.64453125 disk.dm-0.out‚ 31296.828125! disk.dm-1.out‚6558.1943359375 disk.nvme0c0n1.in‚7.1875' disk.nvme1c1n1.out‚15420.8740234375 disk.rbd1.out‚0% disk.nvme1n1.out‚15420.8740234375% disk./.usageGB‚1169.2470779418945 disk.rbd0.in‚0& proc.memory.availableMB‚ 493.421875 cpu‚25.684193177873023 proc.cpu.threads‚17) disk./.usagePercent‚76.16004415840382 disk.md0.in‚0!åìžÓ‚(:“ ­ƒ÷Ò²£ûÞ gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.251' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.75033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âqìs2Ô- _wandbruntime‚4301  _runtime‚4301Öç—>ÉÕà trainer/global_step‚4549 % _timestamp‚1.7845293371448488e+09 _runtime‚4301.413492283 _step‚371 ' lr-AdamW/pg1‚0.00015973153025914026 ' lr-AdamW/pg2‚0.00015973153025914026Îñ‡HÎÖÈ % _timestamp‚1.7845293371448488e+09 _runtime‚4301.413492283 _step‚371 ' lr-AdamW/pg1‚0.00015973153025914026 ' lr-AdamW/pg2‚0.00015973153025914026 trainer/global_step‚4549ó¶u92×- _wandbruntime‚4301  _runtime‚4301•}ÅÖm ØßO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45007163 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36812941 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701331366558720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701331366558720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36812941 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156658593635328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701331725022208 TGDeviceStatsMonitor.on_train_ba“ûxtch_start/allocation.large_pool.allocated‚36812943 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10884827 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44997020 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696050359473664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5273054168430 _step‚372 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151346842735104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚691978899183280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5281024124416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44997020 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696050359473664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5273054168430 _step‚372 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151346842735104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚691978899183280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5281024124416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701331366558720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701331366558720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36812941 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156658593635328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274ôDî™q2Ú- _wandbruntime‚4302  _runtime‚4302¹È*§ð&ÛêM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚697400850008180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10886865 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696198616936448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5282134223360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81837569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5312761143808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45006676 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5274162556682 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7942326 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81837569 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45016833 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701481109630464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10887145 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5274503118068 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18829188 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156692765536768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5282475654656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36820734 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45016833 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7942323 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156692891508224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692126346890112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784529337777813e+09 _step‚373 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36820736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚697400492407434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701480751159808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151380023005184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36820734 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4302.046934045 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696198633975808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5312868503040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36820736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45006676 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5282134223360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚696198616936448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696198616936448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5282134223360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81837569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5312761143808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45006676 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5274162556682 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7942326 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81837569 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45016833 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701481109630464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10887145 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5274503118068 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18829188 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156692765536768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5282475654656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36820734 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45016833 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7942323 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 \º wœHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156692891508224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692126346890112 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784529337777813e+09 _step‚373 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36820736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚697400492407434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701480751159808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151380023005184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36820734 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4302.046934045 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696198633975808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5312868503040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36820736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45006676õ¿/ók2Ý- _wandbruntime‚4302  _runtime‚4302ò.UÞ 2loss/FBt1/assignment_loss‚0.007938358885990766 .loss/FBt1/detection_loss‚0.397231509718692 /loss/FBt2/detection_loss‚0.8983151928405277 %loss/total_loss‚8.578648327805393 epoch‚11 trainer/global_step‚4549 _step‚374 1loss/SRqqt1/assignment_loss‚0.521600256681867 /loss/FRt2/detection_loss‚0.5161867161146999 _runtime‚4302.047161785 1loss/SRqqt2/detection_loss‚1.0143718364083074 $ _timestamp‚1.784529337779008e+09 0loss/FRt1/assignment_loss‚0.7093352586418004 0loss/FRt2/assignment_loss‚1.8234870552429425 2loss/SRqqt2/assignment_loss‚0.6232638444598706 3loss/FBt2/assignment_loss‚0.0008339729136945806 .loss/FRt1/detection_loss‚1.439387585567633 1loss/SRqqt1/detection_loss‚0.6266967403293662ë;•³ ߆ 0loss/FRt1/assignment_loss‚0.7093352586418004 0loss/FRt2/assignment_loss‚1.8234870552429425 2loss/SRqqt2/assignment_loss‚0.6232638444598706 3loss/FBt2/assignment_loss‚0.0008339729136945806 .loss/FRt1/detection_loss‚1.439387585567633 1loss/SRqqt1/detection_loss‚0.6266967403293662 2loss/FBt1/assignment_loss‚0.007938358885990766 .loss/FBt1/detection_loss‚0.397231509718692 /loss/FBt2/detection_loss‚0.8983151928405277 %loss/total_loss‚8.578648327805393 epoch‚11 trainer/global_step‚4549 _step‚374 1loss/SRqqt1/assignment_loss‚0.521600256681867 /loss/FRt2/detection_loss‚0.5161867161146999 _runtime‚4302.047161785 1loss/SRqqt2/detection_loss‚1.0143718364083074 $ _timestamp‚1.784529337779008e+09ö­¦Oà‚(:„ ¼ƒ÷Ò‡þÌÞ# memory_percent‚96.9768762588501) proc.memory.percent‚17.71860122680664 network.sent‚ 7.343585e+06 network.recv‚ 1.121638e+06) disk./.usagePercent‚76.16003595876143% disk./.usageGB‚1169.2469520568848 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚15421.48193359375 disk.md0.out‚ 114.1328125" disk.dm-1.out‚6558.31787109375( disk.nvme1c1n1.out‚15421.48193359375 disk.nvme0c0n1.in‚7.1875% disk.nvme0c0n1.out‚22679.02734375 disk.nvme0n1.in‚7.1875# disk.nvme0n1.out‚22679.02734375 disk.rbd0.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2903.015625 disk.dm-0.in‚ 8.64453125( proc.memory.availableMB‚ 495.30859375 cpu‚25.683492161655064 proc.cpu.threads‚17 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-0.out‚31297.31640625 disk.dm-1.in‚0šÖäÅžá‚(:“ ¼ƒ÷Òó²Ÿß gpu.0.gpu‚75.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.703' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.56766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é"­ažâ‚(:“ ˃÷ÒãòôÝ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚261.089' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.02966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Öx¶‡ã‚(:ü ˃÷ÒÜ÷ŠÞ$ proc.memory.availableMB‚495.0625) proc.memory.percent‚17.71860122680664 proc.cpu.threads‚17) disk./.usagePercent‚76.16003720113149$ disk./.usageGB‚1169.246971130371 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125$ disk.nvme1n1.out‚15423.068359375 disk.dm-0.in‚ 8.64453125 cpu‚25.682806207195192 network.sent‚ 7.395087e+06 network.recv‚ 1.12611e+06 disk.nvme0c0n1.in‚7.1875" disk.nvme0n1.out‚ 22690.3671875 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚15423.068359375 disk.md0.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0 disk.rbd1.out‚0# disk.dm-0.out‚31309.46044921875 disk.dm-1.in‚0" disk.dm-1.out‚6558.41943359375$ memory_percent‚96.97837829589844! proc.memory.rssMB‚ 2903.015625$ disk.nvme0c0n1.out‚ 22690.3671875 disk.md0.out‚ 114.4453125Õ—F•ä‚(:Š Úƒ÷ÒòŽˆÞ( proc.memory.availableMB‚ 495.30859375 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0( disk.nvme1c1n1.out‚15423.89013671875 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚6558.54443359375! proc.memory.rssMB‚ 2903.015625) proc.memory.percent‚17.71860122680664 network.recv‚ 1.129373e+06& disk.nvme0n1.out‚22691.64599609375 disk.rbd0.out‚0 disk.md0.out‚114.8125 disk.nvme0c0n1.in‚7.1875 proc.cpu.threads‚17 network.sent‚ 7.400087e+06) disk./.usagePercent‚76.16007099359709 disk.nvme1c1n1.in‚ 1.45703125# disk.dm-0.out‚31310.63623046875( disk.nvme0c0n1.out‚22691.64599609375 cpu‚25.682003949190324% disk./.usageGB‚1169.2474899291992 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚15423.89013671875 disk.dm-1.in‚0# memory_percent‚96.9768762588501üäÞ¹›å‚(: Úƒ÷Ò¬—ÎÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.66' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.88666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì¹)Í2æ- _wandbruntime‚4346  _runtime‚4346V¤nãÈç ' lr-AdamW/pg1‚0.00015928720332469834 ' lr-AdamW/pg2‚0.00015928720332469834 trainer/global_step‚4599 $ _timestamp‚1.784529382331334e+09 _runtime‚4346.599471871 _step‚375Ív$ÂÍèÇ ' lr-AdamW/pg1‚0.00015928720332469834 ' lr-AdamW/pg2‚0.00015928720332469834 trainer/global_step‚4599 $ _timestamp‚1.784529382331334e+09 _runtime‚4346.599471871 _step‚375÷ ¤v2é- _wandbruntime‚4346  _runtime‚4346Âmãç'êáO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704679327539352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37202593 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37202591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703464067541504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19026709 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5336934650368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10999133 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19026422 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153005856239104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703464067541504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5336934650368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8027292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37202593 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚704678969938718 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5328877943112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚703464084580864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5328537381838 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45491815 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82694408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚708800660767744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703464084580864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5336593226240ãÍØ¢ì'ëæO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19026422 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153005856239104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703464067541504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5336934650368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8027292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37202593 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚704678969938718 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5328877943112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚703464084580864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5328537381838 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45491815 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82694408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚708800660767744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703464084580864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5336593226240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704679327539352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37202593 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37202591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703464067541504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19026709 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5336934650368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10999133 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0øE‡o»2ì- _wandbruntime‚4347  _runtime‚4347¶ú!íëM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45491328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5337703325184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚703612342043648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37210386 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703612359083008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 trainer/global_step‚4599 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19030464 QEDeviceStatsMonitor.on_train_batch_²•V—Ðend/inactive_split.large_pool.freed‚8029023 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5329645770090 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82711871 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37210384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚704827508994442 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8029026 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158407599599616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703612342043648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37210384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚699497863224352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5338044756480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37210386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82701712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚708950403839488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19030755 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845293829658186e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82711871 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45501485 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚699497880263712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153039017896960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5368689061888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703612359083008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚704827866595188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45491328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11001729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11001441 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚377 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45501485 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚4347.234079326 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚4347.234079326 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703612359083008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 trainer/global_step‚4599 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19030464 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8029023 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5329645770090 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82711871 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37210384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚704827508994442 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8029026 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158407599599616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703612342043648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37210384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚699497863224352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5338044756480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37210386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82701712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚708950403839488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19030755 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845293829658186e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82711871 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45501485 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚699497880263712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153039017896960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5368689061888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703612359083008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚704827866595188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45491328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11001729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11001441 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚377ù1ðöñ2ï- _wandbruntime‚4347  _runtime‚4347wHR3ð‚ /loss/FBt1/detection_loss‚0.5291175362759066 trainer/global_step‚4599 _runtime‚4347.234802354 _step‚378 1loss/SRqqt1/detection_loss‚0.5674679898965108 epoch‚11 2loss/SRqqt1/assignment_loss‚0.5465172673616776 0loss/FBt1/assignment_loss‚0.0283691658100636 2loss/FBt2/assignment_loss‚0.017717634915974535 /loss/FRt2/detection_loss‚0.5627618903126375 1loss/SRqqt2/detection_loss‚0.9574058985748661 /loss/FBt2/detection_loss‚0.6063684476791201 %loss/total_loss‚8.247724358646877 % _timestamp‚1.7845293829669666e+09 0loss/FRt1/assignment_loss‚0.6623675207002975 0loss/FRt2/assignment_loss‚1.7564919108871546 2loss/SRqqt2/assignment_loss‚0.5697539415287557 /loss/FRt1/detection_loss‚1.4433851547039116•{«g ñ‡ %loss/total_loss‚8.247724358646877 % _timestamp‚1.7845293829669666e+09 0loss/FRt1/assignment_loss‚0.6623675207002975 0loss/FRt2/assignment_loss‚1.7564919108871546 2loss/SRqqt2/assignment_loss‚0.5697539415287557 /loss/FRt1/detection_loss‚1.4433851547039116 /loss/FBt1/detection_loss‚0.5291175362759066 trainer/global_step‚4599 _runtime‚4347.234802354 _step‚378 1loss/SRqqt1/detection_loss‚0.5674679898965108 epoch‚11 2loss/SRqqt1/assignment_loss‚0.5465172673616776 0loss/FBt1/assignment_loss‚0.0283691658100636 2loss/FBt2/assignment_loss‚0.017717634915974535 /loss/FRt2/detection_loss‚0.5627618903126375 1loss/SRqqt2/detection_loss‚0.9574058985748661 /loss/FBt2/detection_loss‚0.6063684476791201úhØÄCò‚(:‚ éƒ÷ÒÝ¡”Þ' disk.nvme0c0n1.out‚22693.5458984375 disk.rbd0.out‚0! proc.memory.rssMB‚ 2903.015625 cpu‚25.681262988925738 disk.nvme1c1n1.in‚ 1.45703125% disk.nvme1c1n1.out‚15424.44921875 disk.rbd1.out‚0# disk.dm-0.out‚31312.58642578125# memory_percent‚96.9766616821289 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.nvme1n1.out‚15424.44921875 disk.md0.out‚ 115.0234375 disk.dm-1.in‚0% disk.nvme0n1.out‚22693.5458984375) proc.memory.percent‚17.71860122680664 network.recv‚ 1.132678e+06% disk./.usageGB‚1169.2482872009277 disk.nvme1n1.in‚ 1.45703125" disk.dm-1.out‚6558.58349609375 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0% proc.memory.availableMB‚ 495.34375 network.sent‚ 7.415847e+06) disk./.usagePercent‚76.16012292466554 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125[é“›“ó‚(:ˆ éƒ÷ÒÑÞÞÞ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.942' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Lqü’ô‚(:‡ øƒ÷ÒÜŽõÝ network.sent‚ 7.465593e+06) disk./.usagePercent‚76.16012043992542& disk.nvme0n1.out‚22694.33349609375 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚15424.9384765625 disk.rbd1.out‚0! proc.memory.rssMB‚ 2903.015625$ disk./.usageGB‚1169.248249053955 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚31313.13720703125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875( proc.memory.availableMB‚ 495.08203125) proc.memory.percent‚17.71860122680664 network.recv‚ 1.136972e+06 disk.md0.out‚ 115.3203125 disk.dm-1.out‚ 6558.6796875 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚15424.9384765625$ memory_percent‚96.97825908660889( disk.nvme0c0n1.out‚22694.33349609375 disk.nvme1c1n1.in‚ 1.45703125 cpu‚25.680531317532726 proc.cpu.threads‚17û\·ƒõ‚(:’ øƒ÷Ò­Û‡Þ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚119.401' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.800333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 øi†ö‚(:û ‡„÷ÒûÒšÞ! proc.memory.rssMB‚ 2903.015625) proc.memory.percent‚17.71860122680664 network.recv‚ 1.140408e+06) disk./.usagePercent‚76.16012168229548 disk.md0.out‚ 115.4296875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125! disk.nvme1n1.out‚ 15425.453125& proc.memory.availableMB‚ 495.078125 disk.nvme0n1.in‚7.1875& disk.nvme0n1.out‚22696.36962890625 network.sent‚ 7.470449e+06 disk.dm-1.out‚ 6558.71875 disk.nvme0c0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd1.out‚0# memory_percent‚96.9782829284668( disk.nvme0c0n1.out‚22696.36962890625# disk.nvme1c1n1.out‚ 15425.453125 cpu‚25.679684328435545 proc.cpu.threads‚17% disk./.usageGB‚1169.2482681274414 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚31315.39306640625 disk.dm-1.in‚0 disk.rbd1.in‚0^·Åfž÷‚(:“ ‡„÷ÒÔ¦ÈÞ gpu.0.gpu‚63.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚259.747' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.58233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚099¼52ø- _wandbruntime‚4392  _runtime‚4392ÙeÆùÀ trainer/global_step‚4649 % _timestamp‚1.7845294280204217e+09 _runtime‚ 4392.28857392 _step‚379 & lr-AdamW/pg1‚0.0001588383870362853 & lr-AdamW/pg2‚0.0001588383870362853¼É¡ÁËúÅ % _timestamp‚1.7845294280204217e+09 _runtime‚ 4392.28857392 _step‚379 & lr-AdamW/pg1‚0.0001588383870362853 & lr-AdamW/pg2‚0.0001588383870362853 trainer/global_step‚4649ûMN%2û- _wandbruntime‚4392  _runtime‚4392Név!üáO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚716270313440256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37592243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚706721982969840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845294280215995e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11113392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11113098 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5423405066752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160088155122688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154664851130880 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚380 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83558565 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚716270313440256 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5384361156520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710877809688064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5392503752192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37592243 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8113989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.curren ˜¶æÌt‚ 340561274 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚716269954976768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5392162328064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710877809688064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83568710 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45976467 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45966324 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5392503752192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19227087 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5423303991808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5384020595246+ù°Xì'ýæO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37592241 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚712105986525726 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154664851130880 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚380 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83558565 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚716270313440256 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5384361156520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710877809688064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5392503752192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37592243 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8113989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚716269954976768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5392162328064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710877809688064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83568710 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45976467 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45966324 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5392503752192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19227087 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5423303991808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5384020595246 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚716270313440256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37592243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚706721982969840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845294280215995e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11113392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11113098 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5423405066752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160088155122688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37600036 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚706869396597952 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5385469544884 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45986137 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19231085 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154698050013184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83576014 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚381 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83586173 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37600034 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83576014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8115723 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160122450012160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19231376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5393272427008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚716419339577856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8115726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845294286533823e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5385128983498 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5393272427008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37600036 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45975980 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚711026084190208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37600034 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11115650 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11115362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154698031400960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5424521776128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚716419698048512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45975980 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5393613858304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚706869413637312 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚4649 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83586173 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚711026067150848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5424418611200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160122571789312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚712254525581450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4392.921673823Øàç½ö&€ðM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5393272427008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37600036 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45975980 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚711026084190208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37600034 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11115650 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11115362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154698031400960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5424521776128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚716419698048512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45975980 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5393613858304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚706869413637312 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚4649 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83586173 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚711026067150848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5424418611200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160122571789312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚712254525581450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4392.921673823 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45986137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37600036 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚706869396597952 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5385469544884 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45986137 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19231085 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154698050013184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83576014 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚381 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83586173 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37600034 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83576014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8115723 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160122450012160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19231376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5393272427008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚716419339577856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8115726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845294286533823e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5385128983498ý¡W 2- _wandbruntime‚4392  _runtime‚4392·ÅÌ‚ÿ 0loss/FRt2/assignment_loss‚1.8891032478388619 2loss/FBt2/assignment_loss‚0.006265238282703824 1loss/SRqqt2/detection_loss‚0.9090166025465549 .loss/FBt1/detection_loss‚0.550406804200835 _step‚382 0loss/FRt1/assignment_loss‚0.7677542923883414 /loss/FRt2/detection_loss‚0.5210861381557128 1loss/SRqqt1/detection_loss‚0.6083619649105362 /loss/FBt2/detection_loss‚0.7096817781432316 %loss/total_loss‚8.447120421402225 2loss/SRqqt1/assignment_loss‚0.5044148817684124 1loss/FBt1/assignment_loss‚0.01486538759287731 epoch‚11 _runtime‚4392.922519855 2loss/SRqqt2/assignment_loss‚0.5379762761250714 /loss/FRt1/detection_loss‚1.4281878094490867 train~¦€9er/global_step‚4649 " _timestamp‚1.7845294286546e+09;I\> ƒ„ 2loss/SRqqt2/assignment_loss‚0.5379762761250714 /loss/FRt1/detection_loss‚1.4281878094490867 trainer/global_step‚4649 " _timestamp‚1.7845294286546e+09 0loss/FRt2/assignment_loss‚1.8891032478388619 2loss/FBt2/assignment_loss‚0.006265238282703824 1loss/SRqqt2/detection_loss‚0.9090166025465549 .loss/FBt1/detection_loss‚0.550406804200835 _step‚382 0loss/FRt1/assignment_loss‚0.7677542923883414 /loss/FRt2/detection_loss‚0.5210861381557128 1loss/SRqqt1/detection_loss‚0.6083619649105362 /loss/FBt2/detection_loss‚0.7096817781432316 %loss/total_loss‚8.447120421402225 2loss/SRqqt1/assignment_loss‚0.5044148817684124 1loss/FBt1/assignment_loss‚0.01486538759287731 epoch‚11 _runtime‚4392.922519855þI®§¾—„‚(:Œ –„÷ÒÞξÞ disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0& disk.nvme0n1.out‚22697.55322265625! proc.memory.rssMB‚ 2903.015625 cpu‚25.678899862870995 proc.cpu.threads‚17 network.recv‚ 1.14395e+06( disk.nvme1c1n1.out‚15426.06298828125& disk.nvme1n1.out‚15426.06298828125# disk.dm-0.out‚31316.20556640625( disk.nvme0c0n1.out‚22697.55322265625 disk.rbd0.in‚0$ memory_percent‚96.97716236114502( proc.memory.availableMB‚ 495.26171875 network.sent‚ 7.488199e+06 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 disk.md0.out‚ 115.828125 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0) proc.memory.percent‚17.71860122680664% disk./.usageGB‚1169.2492408752441 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚6558.82470703125 disk.nvme0n1.in‚7.1875) disk./.usagePercent‚76.16018504316848ÔSgŸ…‚(:” –„÷ÒåÒß gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.085' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.361666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØN˜†‚(: ¥„÷Ò˜¨ƒÞ( disk.nvme1c1n1.out‚15426.79052734375 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125" disk.dm-1.out‚6558.92236328125 disk.nvme0c0n1.in‚7.1875$ memory_percent‚96.97871208190918 network.recv‚ 1.148402e+06) disk./.usagePercent‚76.16018554011652 disk.rbd1.out‚0 disk.md0.in‚0( disk.nvme0c0n1.out‚22699.58740234375 disk.nvme0n1.in‚7.1875! proc.memory.rssMB‚ 2903.015625 cpu‚25.678129826445428 network.sent‚ 7.537686e+06% disk./.usageGB‚1169.2492485046387 disk.md0.out‚ 116.2109375 disk.dm-0.in‚ 8.64453125 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme1n1.out‚15426.79052734375# disk.dm-0.out‚31318.04736328125 disk.dm-1.in‚0& disk.nvme0n1.out‚22699.58740234375' proc.memory.availableMB‚ 495.0078125) proc.memory.percent‚17.71860122680664 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125g‚”&“‡‚(:ˆ ¥„÷ÒÞÅŠÞ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚258.813' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚86.271 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎcïÉŒˆ‚(: ´„÷ÒŽ¦•Þ! proc.memory.rssMB‚ 2903.015625 cpu‚25.677413923159943 network.recv‚ 1.151757e+06# disk.nvme1c1n1.out‚ 15427.546875! disk.nvme1n1.out‚ 15427.546875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0) proc.memory.percent‚17.71860122680664 network.sent‚ 7.542707e+06% disk./.usageGB‚1169.2492141723633 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0" disk.dm-1.out‚6559.01318359375' disk.nvme0c0n1.out‚22700.7919921875' proc.memory.availableMB‚ 495.0078125( disk./.usagePercent‚76.1601833038504 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚31318.88330078125% disk.nvme0n1.out‚22700.7919921875 disk.rbd0.out‚0$ memory_percent‚96.97871208190918 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.out‚ 116.69140625Ç{Dž‰‚(:“ ´„÷Ò—¦ÏÞ gpu.0.gpu‚60.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.219' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.40633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ÝË2Š- _wandbruntime‚4437  _runtime‚4437_Q-”É‹Ã ' lr-AdamW/pg2‚0.00015838511022053277 trainer/global_step‚4699 % _timestamp‚1.7845294731120937e+09 _runtime‚4437.380201106 _step‚383 ' lr-AdamW/pg1‚0.00015838511022053277>É?ÎŒÈ trainer/global_step‚4699 % _timestamp‚1.7845294731120937e+09 _runtime‚4437.380201106 _step‚383 ' lr-AdamW/pg1‚0.00015838511022053277 ' lr-AdamW/pg2‚0.00015838511022053277ÿä`·Ñ2- _wandbruntime‚4437  _runtime‚4437©xé©ç'ŽáO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718291517755904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11227903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5448072854016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37981891 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156323883247104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156323864634880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37981891 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719533003112734 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37981893 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _step‚384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719533360713368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4437.384614404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8200692 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46450976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8200689 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5448072854016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5479225918976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚723739607649280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718291517755904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 trainer/global_step‚4699 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723739249185792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46461119 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5439503808654 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5447731429888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19428595 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37981893 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46450976 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4437.384614404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8200692 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46450976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8200689 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5448072854016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5479225918976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚723739607649280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718291517755904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 trainer/global_step‚4699 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723739249185792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46461119 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5439503808654 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5447731429888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19428595 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0€«%dÛ2- _wandbruntime‚4438  _runtime‚4438¸¹9‘ëM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84450316 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19432342 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84450316 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5449182960128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718439792258048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718439809297408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚723888992257536 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚719681899769204 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8202426 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84460475 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚385 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 trainer/global_step‚4699 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5448841528832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156357063517184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46460632 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46470789 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161837401349632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156357044904960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5480337832448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5440952758292 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5440612196906 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37989686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37989686 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272'J/  WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚723888992257536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84460475 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19432625 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5480222084608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37989684 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46470789 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5449182960128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11230199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚4438.010835096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚714240947010912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚714240929971552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11229919 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719681542168458 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19432342 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84450316 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5449182960128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718439792258048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718439809297408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚723888992257536 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚719681899769204 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8202426 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84460475 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚385 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 trainer/global_step‚4699 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5448841528832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156357063517184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46460632 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46470789 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161837401349632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156357044904960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5480337832448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5440952758292 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5440612196906 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37989686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37989686 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚723888992257536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84460475 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19432625 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5480222084608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37989684 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46470789 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5449182960128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11230199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚4438.010835096ßRý‘2“- _wandbruntime‚4438  _runtime‚4438ú>öš”‚ 0loss/FRt2/assignment_loss‚1.9063965103371812 1loss/SRqqt2/detection_loss‚0.9848875041851508 trainer/global_step‚4699 /loss/FBt2/detection_loss‚1.0561754460490884 epoch‚11 2loss/SRqqt1/assignment_loss‚0.5848426741097045 2loss/SRqqt2/assignment_loss‚0.5829358763182036 /loss/FRt1/detection_loss‚1.4591417971824447 /loss/FBt1/detection_loss‚0.4599024190827122 2loss/FBt2/assignment_loss‚0.005535295934026051 %loss/total_loss‚9.090736941901305 $ _timestamp‚1.784529473743674e+09 _runtime‚4438.011591707 _step‚386 0loss/FRt1/assignment_loss‚0.8218414319042441 2loss/FBt1/assignment_loss‚0.016196806852467063 /loss/FRt2/detection_loss‚0.5377236193175474 0loss/SRqqt1/detection_loss‚0.675157560628535ü8_¶ •‡ epoch‚11 2loss/SRqqt1/assignment_loss‚0.5848426741097045 2loss/SRqqt2/assignment_loss‚0.5829358763182036 /loss/FRt1/detection_loss‚1.4591417971824447 /loss/FBt1/detection_loss‚0.4599024190827122 2loss/FBt2/assignment_loss‚0.005535295934026051 %loss/total_loss‚9.090736941901305 $ _timestamp‚1.784529473743674e+09 _runtime‚4438.011591707 _step‚386 0loss/FRt1/assignment_loss‚0.8218414319042441 2loss/FBt1/assignment_loss‚0.016196806852467063 /loss/FRt2/detection_loss‚0.5377236193175474 0loss/SRqqt1/detection_loss‚0.675157560628535 0loss/FRt2/assignment_loss‚1.9063965103371812 1loss/SRqqt2/detection_loss‚0.9848875041851508 trainer/global_step‚4699 /loss/FBt2/detection_loss‚1.0561754460490884‚›Ü˜–‚(:‚ Ä÷Ò‡‚ÃÞ% disk./.usageGB‚1169.0115585327148 disk.rbd1.out‚0 disk.md0.in‚0 disk.md0.out‚ 116.921875 disk.dm-0.out‚ 31323.84375! proc.memory.rssMB‚ 2903.015625) proc.memory.percent‚17.71860122680664% disk.nvme0n1.out‚22705.7744140625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚15428.07666015625 disk.nvme1n1.in‚ 1.45703125" disk.dm-1.out‚6559.06005859375$ memory_percent‚96.97742462158203% proc.memory.availableMB‚ 495.21875 disk.nvme0c0n1.in‚7.1875 disk.rbd0.out‚0 cpu‚25.676644146510434 proc.cpu.threads‚17 network.recv‚ 1.155413e+06) disk./.usagePercent‚76.14470337291743' disk.nvme0c0n1.out‚22705.7744140625 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0& disk.nvme1n1.out‚15428.07666015625 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 network.sent‚ 7.560402e+06¢OåOž—‚(:“ Ä÷Ò°üÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.844' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.28133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠB˜Œ˜‚(: Ò„÷Ò¸¬Þ disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.97897434234619 proc.cpu.threads‚17 network.recv‚ 1.159466e+06 disk.dm-0.out‚31324.43359375 disk.dm-1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 494.96484375) proc.memory.percent‚17.71860122680664 disk.md0.out‚ 117.453125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875$ disk.nvme0n1.out‚22706.869140625 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 cpu‚25.675830103051823$ disk./.usageGB‚1169.011775970459 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125! disk.dm-1.out‚6559.2431640625' disk.nvme1c1n1.out‚15428.8818359375% disk.nvme1n1.out‚15428.8818359375! proc.memory.rssMB‚ 2903.015625 network.sent‚ 7.611887e+06( disk./.usagePercent‚76.1447175359361& disk.nvme0c0n1.out‚22706.869140625 disk.rbd0.in‚0 disk.rbd0.out‚0眙‚(:‘ Ò„÷ÒàÏöÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.177' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.39233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÌŠt“š‚(:ˆ á„÷Ò¡§²Þ( proc.memory.availableMB‚ 494.96484375) proc.memory.percent‚17.71860122680664 disk.md0.in‚0! disk.dm-1.out‚6559.3134765625& disk.nvme1n1.out‚15430.00048828125! proc.memory.rssMB‚ 2903.015625 proc.cpu.threads‚17) disk./.usagePercent‚76.14883425336303% disk./.usageGB‚1169.0749778747559 disk.md0.out‚ 117.7734375 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0( disk.nvme1c1n1.out‚15430.00048828125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125 network.sent‚ 7.616746e+06 disk.dm-0.in‚ 8.64453125( disk.nvme0c0n1.out‚22736.41357421875& disk.nvme0n1.out‚22736.41357421875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 cpu‚25.675018692480815 network.recv‚ 1.162861e+06 disk.dm-0.out‚ 31354.328125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875$ memory_percent‚96.97897434234619ºk!Ež›‚(:“ á„÷Ò§­ëÞ gpu.0.gpu‚32.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.359' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.11966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Då¨ 2œ- _wandbruntime‚4482  _runtime‚4482$çZÉà ' lr-AdamW/pg1‚0.00015792740199056346 ' lr-AdamW/pg2‚0.00015792740199056346 trainer/global_step‚4749 % _timestamp‚1.7845295185274928e+09 _runtime‚4482.797157935 _step‚387!”MÎžÈ _runtime‚4482.797157935 _step‚387 ' lr-AdamW/pg1‚0.00015792740199056346 ' lr-AdamW/pg2‚0.00015792740199056346 trainer/global_step‚4749 % _timestamp‚1.7845295185274928e+09ƒ0’1j2Ÿ- _wandbruntime‚4482  _runtime‚4482îÿÍŽæ' àO _step‚388 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85307169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5503300531712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚721465049717040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721465032677680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5494987022062 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731208543394816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725705242863104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46945771 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38371541 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8287392 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚726960019699742 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725705259902464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5534985891840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38371543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19629496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46945771 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725705259902464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163517752470016 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8287389 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11342397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5503641955840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5503300531712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5534874331136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46935628 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163517882642944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38371541 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚4749 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5503641955840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452—G’<¸ ¡åO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5534874331136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46935628 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163517882642944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38371541 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚4749 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5503641955840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚388 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85307169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5503300531712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚721465049717040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721465032677680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5494987022062 MGDeviceStatsMonitor.on_tr’ì¡Ì3ain_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731208543394816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725705242863104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46945771 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38371541 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8287392 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚726960019699742 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725705259902464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5534985891840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38371543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19629496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46945771 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725705259902464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163517752470016 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8287389 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11342397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5503641955840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5503300531712„¤¬t2¢- _wandbruntime‚4483  _runtime‚4483¾>ð&£êM I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11344384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845295191565576e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5496095410314 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38379334 _runtime‚ 4483.42491452 _step‚389 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38379334 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731358286466560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727108558755466 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725853534404608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8289123 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚727108916356212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85334777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731358286466560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11344661 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 119942144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85324618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5504410630656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38379336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚721612463345152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731357927995904 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46945284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38379336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85334777 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46945284 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163552041920512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚725853534404608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5504410630656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 138554368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5496435971700 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46955441 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5536103453696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5504752061952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5504752061952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚721612480384512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163552180474880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19633507 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158016077021184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5535983511552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ËQ(­õ&¤ïM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46945284 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163552041920512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚725853534404608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5504410630656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 138554368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5496435971700 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46955441 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5536103453696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5504752061952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5504752061952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚721612480384512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163552180474880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19633507 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158016077021184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5535983511552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11344384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845295191565576e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5496095410314 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38379334 _runtime‚ 4483.42491452 _step‚389 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38379334 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731358286466560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727108558755466 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725853534404608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8289123 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚727108916356212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85334777 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731358286466560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11344661 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 119942144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85324618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5504410630656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38379336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚721612463345152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731357927995904 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46945284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38379336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85334777 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360…_ÎÆn2¥- _wandbruntime‚4483  _runtime‚4483¥rÌ ¦„ trainer/global_step‚4749 0loss/FRt2/assignment_loss‚1.8487761407101302 3loss/SRqqt2/assignment_loss‚0.48374429069382097 % _timestamp‚1.7845295191578279e+09 0loss/FRt1/assignment_loss‚0.7105502343374835 1loss/SRqqt1/detection_loss‚0.6024222261746125 %loss/total_loss‚9.037468144104203 _runtime‚4483.425783791 _step‚390 2loss/SRqqt1/assignment_loss‚0.5046357930088493 2loss/FBt1/assignment_loss‚0.027437120572807634 3loss/FBt2/assignment_loss‚0.0002574469957459729 /loss/FRt2/detection_loss‚0.5538090733944989 1loss/SRqqt2/detection_loss‚1.0790509891677138 .loss/FRt1/detection_loss‚1.417593144996415 .loss/FBt1/detection_loss‚0.523790900166043 /loss/FBt2/detection_loss‚1.2854007838860801 epoch‚11û5QM§‰ 0loss/FRt1/assignment_loss‚0.7105502343374835 1loss/SRqqt1/detection_loss‚0.6024222261746125 %loss/total_loss‚9.037468144104203 _runtime‚4483.425783791 _step‚390 2loss/SRqqt1/assignment_loss‚0.5046357930088493 2loss/FBt1/assignment_loss‚0.027437120572807634 3loss/FBt2/assignment_loss‚0.0002574469957459729 /loss/FRt2/detection_loss‚0.5538090733944989 1loss/SRqqt2/detection_loss‚1.0790509891677138 .loss/FRt1/detection_loss‚1.417593144996415 .loss/FBt1/detection_loss‚0.523790900166043 /loss/FBt2/detection_loss‚1.2854007838860801 epoch‚11 trainer/global_step‚4749 0loss/FRt2/assignment_loss‚1.8487761407101302 3loss/SRqqt2/assignment_loss‚0.48374429069382097 % _timestamp‚1.7845295191578279e+09†‹!â铨‚(:ˆ ð„÷Ò ©½Þ disk.md0.in‚0 disk.md0.out‚ 118.24609375 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚31586.27197265625$ memory_percent‚96.96884155273438 network.sent‚ 7.632654e+06 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.71860122680664 proc.cpu.threads‚17" disk.dm-1.out‚7094.54345703125 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚22946.03564453125 disk.nvme0n1.in‚7.1875( disk.nvme1c1n1.out‚15988.55712890625 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125! proc.memory.rssMB‚ 2903.015625& disk.nvme0n1.out‚22946.03564453125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0& disk.nvme1n1.out‚15988.55712890625# proc.memory.availableMB‚496.625 cpu‚25.67432339046237 network.recv‚ 1.16617e+06) disk./.usagePercent‚76.14474958908362% disk./.usageGB‚1169.0122680664062¡HYž©‚(:“ ð„÷ÒóÂüÞ gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.264' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.42133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x§û窂(:… ÿ„÷Ò¯Ù‘Þ disk.md0.out‚ 118.7890625" disk.dm-1.out‚7094.62548828125# disk.nvme0n1.out‚22956.58984375( disk.nvme1c1n1.out‚15989.65185546875 disk.rbd1.in‚0& disk.nvme1n1.out‚15989.65185546875( proc.memory.availableMB‚ 496.59765625 disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚31596.6884765625 disk.nvme0c0n1.in‚7.1875 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 proc.cpu.threads‚17 network.recv‚ 1.170012e+06 disk.md0.in‚0% disk.nvme0c0n1.out‚22956.58984375$ memory_percent‚96.96900844573975! proc.memory.rssMB‚ 2903.015625) proc.memory.percent‚17.71860122680664 network.sent‚ 7.681996e+06( disk./.usagePercent‚76.0578850623626 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 cpu‚25.67358241204226% disk./.usageGB‚1167.6786804199219håÔKž«‚(:“ ÿ„÷Ò‡Ô›Þ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚232.765' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.58833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0khžc¬‚(:„ Ž…÷Òû•‡Þ% disk./.usageGB‚1167.6786041259766 disk.nvme0c0n1.in‚7.1875 disk.rbd1.out‚0& disk.nvme1n1.out‚15990.58154296875# memory_percent‚96.9675064086914) proc.memory.percent‚17.71860122680664 cpu‚25.672788089693867) disk./.usagePercent‚76.05788009288237 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125! proc.memory.rssMB‚ 2903.015625' disk.nvme0c0n1.out‚22958.6513671875 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0% proc.memory.availableMB‚ 496.84375 proc.cpu.threads‚17 network.recv‚ 1.173102e+06 disk.md0.in‚0 disk.md0.out‚ 119.546875 disk.dm-0.out‚31598.05078125" disk.dm-1.out‚7094.67626953125% disk.nvme0n1.out‚22958.6513671875 disk.rbd0.out‚0( disk.nvme1c1n1.out‚15990.58154296875 network.sent‚ 7.688062e+06VEÌœ­‚(:‘ Ž…÷ÒÑÕÆÞ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.9' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.63333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³x+ 2®- _wandbruntime‚4528  _runtime‚4528¬ÔF¯À & lr-AdamW/pg1‚0.0001574652917441218 & lr-AdamW/pg2‚0.000­çÈ€1574652917441218 trainer/global_step‚4799 $ _timestamp‚1.784529563959526e+09 _runtime‚4528.227702194 _step‚391ȈÍ9Ë°Å _step‚391 & lr-AdamW/pg1‚0.0001574652917441218 & lr-AdamW/pg2‚0.0001574652917441218 trainer/global_step‚4799 $ _timestamp‚1.784529563959526e+09 _runtime‚4528.227702194‡+.2±- _wandbruntime‚4528  _runtime‚4528>mÿ€æ'²àO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47430423 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38761193 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738677837603840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38761193 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165232634195456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚733118985009664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5590847821824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5550470235470 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86181471 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚738678196067328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚738678196067328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚738677837603840 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚734387036286750 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5590742552576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 $ _timestamp‚1.784529563961007e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165232758076928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159641891642880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚728836566051280 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38761191 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738677837603840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38761193 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165232634195456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚733118985009664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5590847821824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5550470235470 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86181471 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚738678196067328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚738678196067328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚738677837603840 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚734387036286750 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5590742552576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 $ _timestamp‚1.784529563961007e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165232758076928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159641891642880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚728836566051280 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38761191 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560321163776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784529564590797e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5559979732480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚738827222204928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733267259511808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚733267242472448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8375823 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚4799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8375826 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734535575342474 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38768984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86209079 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47440093 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚728984013758112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5551578623722 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86209079 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560321163776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19834107 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159675071912960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38768986 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚733267242472448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86198920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11458284 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚393 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47429936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5591957310464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚728983996718752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38768986 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734535932943220 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159675090525184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19834392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5591860436992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86198920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚4528.859099438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38768984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47429936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733267259511808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚738827580675584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5551919185108 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560321163776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784529564590797e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5559979732480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚738827222204928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end ¬ÞÅ›/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733267259511808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚733267242472448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8375823 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚4799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8375826 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734535575342474 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38768984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86209079 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47440093 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚728984013758112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5551578623722 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86209079 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560321163776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19834107 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159675071912960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38768986 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚733267242472448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86198920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11458284 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚393 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47429936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5591957310464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚728983996718752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38768986 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734535932943220 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159675090525184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19834392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5591860436992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86198920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚4528.859099438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38768984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47429936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733267259511808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚738827580675584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0‰Ãæ+»2·- _wandbruntime‚4528  _runtime‚4528„æ…= ¸„ 0loss/FRt1/assignment_loss‚0.6537004746984707 .loss/FRt1/detection_loss‚1.508270090171868 1loss/SRqqt1/detection_loss‚0.5904098095735602 /loss/FBt2/detection_loss‚0.5627327342404098 epoch‚11 _step‚394 1loss/SRqqt2/detection_loss‚0.9234644180896585 trainer/global_step‚4799 % _timestamp‚1.7845295645922983e+09 0loss/FRt2/assignment_loss‚1.8408852332635102 2loss/SRqqt1/assignment_loss‚0.4885343428360018 3loss/FBt2/assignment_loss‚0.0021903245276587036 2loss/SRqqt2/assignment_loss‚0.6141395547807973 2loss/FBt1/assignment_loss‚0.014152735951767286 /loss/FRt2/detection_loss‚0.5057286436490932 /loss/FBt1/detection_loss‚0.3880472733855615 %loss/total_loss‚8.092255635168357 _runtime‚4528.860187069ˆGŒ7¹‰ 0loss/FRt1/assignment_loss‚0.6537004746984707 .loss/FRt1/detection_loss‚1.508270090171868 1loss/SRqqt1/detection_loss‚0.5904098095735602 /loss/FBt2/detection_loss‚0.5627327342404098 epoch‚11 _step‚394 1loss/SRqqt2/detection_loss‚0.9234644180896585 trainer/global_step‚4799 % _timestamp‚1.7845295645922983e+09 0loss/FRt2/assignment_loss‚1.8408852332635102 2loss/SRqqt1/assignment_loss‚0.4885343428360018 3loss/FBt2/assignment_loss‚0.0021903245276587036 2loss/SRqqt2/assignment_loss‚0.6141395547807973 2loss/FBt1/assignment_loss‚0.014152735951767286 /loss/FRt2/detection_loss‚0.5057286436490932 /loss/FBt1/detection_loss‚0.3880472733855615 %loss/total_loss‚8.092255635168357 _runtime‚4528.860187069Šwg1©“º‚(:ˆ …÷ÒЯõÝ disk.md0.out‚ 120.00390625" disk.dm-1.out‚7094.89892578125% disk.nvme0c0n1.out‚22981.91015625# disk.nvme0n1.out‚22981.91015625 disk.rbd0.in‚0( proc.memory.availableMB‚ 537.95703125 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0 network.recv‚ 1.176698e+06) disk./.usagePercent‚76.05779909035452 disk.rbd1.in‚0 disk.rbd1.out‚0& disk.nvme1n1.out‚15992.98681640625 disk.md0.in‚0 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0( disk.nvme1c1n1.out‚15992.98681640625$ memory_percent‚96.71657085418701 cpu‚25.672494599168463 proc.cpu.threads‚17! disk.dm-0.out‚31622.509765625 network.sent‚ 7.691205e+06$ disk./.usageGB‚1167.677360534668 disk.nvme1c1n1.in‚ 1.45703125" proc.memory.rssMB‚ 2861.8828125* proc.memory.percent‚17.467546463012695 disk.nvme1n1.in‚ 1.45703125Øx¥»‚(:„ …÷Ò£Ó˜à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.16' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.72 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00hk¼‚(:… ¬…÷ÒÔí¨Þ( disk.nvme1c1n1.out‚15993.65380859375 disk.rbd1.in‚0) disk./.usagePercent‚76.05780530220483% disk./.usageGB‚1167.6774559020996" disk.dm-1.out‚7095.03173828125' disk.nvme0c0n1.out‚22982.8154296875 disk.rbd0.in‚0# memory_percent‚96.9517707824707& proc.memory.availableMB‚ 499.421875 proc.memory.rssMB‚2900.375 disk.dm-1.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.180565e+06 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0 disk.dm-0.in‚ 8.64453125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875* proc.memory.percent‚17.702484130859375 cpu‚25.680034264238923 network.sent‚ 7.755411e+06& disk.nvme1n1.out‚15993.65380859375 disk.md0.out‚ 120.4296875! disk.dm-0.out‚31623.052734375% disk.nvme0n1.out‚22982.8154296875 disk.rbd0.out‚0ýE´È“½‚(:ˆ ¬…÷ÒøùäÞ gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.449' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.483 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž|ý2¾- _wandbruntime‚4545  _runtime‚4545 › ¦°=¿ª{ 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5447369813919067 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8502267003059387 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09601297974586487 )jet/accuracy_1_of_2‚0.785603940486908 *jet/accuracy_1_of_3‚0.9157877564430237 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24464212357997894 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018860546872019768 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28837665915489197 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19696763157844543 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8592941761016846 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7062308192253113 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5191667079925537 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24033012986183167 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3060360848903656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976153373718262 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7304084300994873 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_2_of_4‚0.8064113855361938 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861609935760498 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9369828104972839 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5013811588287354 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 epoch‚11 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6901566982269287 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1308143585920334 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12246756255626678 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894712567329407 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6985105872154236 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019351433962583542 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7603909373283386 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944268465042114 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8551398515701294 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22036448121070862 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 _step‚395 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983856678009033 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38262271881103516 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6729404330253601 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852938055992126 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6101232171058655 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02127324976027012 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9819256663322449 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17938213050365448 *jet/accuracy_1_of_4‚0.9692670106887817 +particle/sensitivity‚0.6366777420043945 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22002151608467102 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.490304172039032 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7827686071395874 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3524053692817688 (loss/val_total_loss‚8.37874780326307 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46488943696022034 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7498883605003357 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995421826839447 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25518155097961426 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46344780921936035 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018860546872019768 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2956885099411011 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.640116274356842 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6870629787445068 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7819334864616394 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10934922844171524 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25133803486824036 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6493942141532898 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6258543729782104 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6400769948959351 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917798638343811 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7284438014030457 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 'particle/f_score‚0.6479290723800659 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6343441605567932 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.613262414932251 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833730697632 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6727108955383301 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +jet/accuracy_4_of_4‚0.01628045178949833 6validation_average_jet_accuracy‚0.3890281021595001 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32216915488243103 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7678473591804504 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2226523905992508 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.296994686126709 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8592941761016846 jet/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7690287828445435 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8586734533309937 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952129125595093 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35085543990135193 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9101406335830688 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30656200647354126 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974170923233032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.056777965277433395 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6742017865180969 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6591784358024597 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.296994686126709 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28066542744636536 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6384710073471069 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23183520138263702 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_5‚1 trainer/global_step‚4799 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_2_of_2‚0.1608104109764099 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7436569929122925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7168099284172058 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8957599401473999 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.640116274356842 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046289920806884766 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7316163778305054 +jet/accuracy_3_of_3‚0.07467396557331085 +jet/accuracy_3_of_4‚0.08696750551462173 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9243221282958984 particle/accuracy_2_of_4‚1 /particle/accuracy_4_of_4‚0.9457802772521973 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6509581804275513 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8838762640953064 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9909628033638 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7134230136871338 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3562459349632263 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4877031445503235 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04980572685599327 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8406273126602173 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7845295815217197e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996184229850769 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22424891591072083 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37729254364967346 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6396082043647766 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3068169057369232 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2964252531528473 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5642975568771362 (particle/accuracy‚0.8514113426208496 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 _runtime‚4545.790293985 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918533563613892 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5022516846656799 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32145434617996216 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940679669380188 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7736878991127014 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740740299224854 jet/accuracy_4_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974170923233032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6881788969039917 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.661471962928772 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8745644688606262 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6912821531295776 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03409580513834953 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6074349284172058 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20540860295295715 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14683854579925537 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5525078773498535 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13807037472724915 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7101224064826965 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8391534090042114 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021332889795303345 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981808662414551 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020076410844922066 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7163493633270264 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1k†£¦À¯{ =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981808662414551 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020076410844922066 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7163493633270264 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5447369813919067 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8502267003059387 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09601297974586487 )jet/accuracy_1_of_2‚0.785603940486908 *jet/accuracy_1_of_3‚0.9157877564430237 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24464212357997894 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018860546872019768 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28837665915489197 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19696763157844543 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8592941761016846 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7062308192253113 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5191667079925537 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24033012986183167 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3060360848903656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976153373718262 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7304084300994873 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_2_of_4‚0.8064113855361938 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861609935760498 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9369828104972839 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5013811588287354 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 epoch‚11 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6901566982269287 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1308143585920334 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12246756255626678 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894712567329407 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6985105872154236 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019351433962583542 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7603909373283386 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944268465042114 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8551398515701294 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22036448121070862 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 _step‚395 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983856678009033 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38262271881103516 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6729404330253601 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852938055992126 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6101232171058655 þÜ“,:"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02127324976027012 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9819256663322449 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17938213050365448 *jet/accuracy_1_of_4‚0.9692670106887817 +particle/sensitivity‚0.6366777420043945 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22002151608467102 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.490304172039032 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7827686071395874 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3524053692817688 (loss/val_total_loss‚8.37874780326307 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46488943696022034 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7498883605003357 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995421826839447 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25518155097961426 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46344780921936035 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018860546872019768 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2956885099411011 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.640116274356842 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6870629787445068 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7819334864616394 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10934922844171524 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25133803486824036 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6493942141532898 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6258543729782104 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6400769948959351 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917798638343811 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7284438014030457 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 'particle/f_score‚0.6479290723800659 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6343441605567932 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.613262414932251 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833730697632 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6727108955383301 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +jet/accuracy_4_of_4‚0.01628045178949833 6validation_average_jet_accuracy‚0.3890281021595001 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32216915488243103 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7678473591804504 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2226523905992508 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.296994686126709 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8592941761016846 jet/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7690287828445435 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8586734533309937 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952129125595093 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35085543990135193 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9101406335830688 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30656200647354126 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974170923233032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.056777965277433395 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6742017865180969 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6591784358024597 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.296994686126709 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28066542744636536 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6384710073471069 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23183520138263702 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_5‚1 trainer/global_step‚4799 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_2_of_2‚0.1608104109764099 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7436569929122925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7168099284172058 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8957599401473999 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.640116274356842 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046289920806884766 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7316163778305054 +jet/accuracy_3_of_3‚0.07467396557331085 +jet/accuracy_3_of_4‚0.08696750551462173 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9243221282958984 particle/accuracy_2_of_4‚1 /particle/accuracy_4_of_4‚0.9457802772521973 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6509581804275513 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8838762640953064 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9909628033638 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7134230136871338 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3562459349632263 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4877031445503235 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04980572685599327 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8406273126602173 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7845295815217197e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996184229850769 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22424891591072083 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37729254364967346 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6396082043647766 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3068169057369232 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2964252531528473 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5642975568771362 (particle/accuracy‚0.8514113426208496 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 _runtime‚4545.790293985 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918533563613892 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5022516846656799 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32145434617996216 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940679669380188 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7736878991127014 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740740299224854 jet/accuracy_4_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974170923233032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6881788969039917 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.661471962928772 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8745644688606262 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6912821531295776 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03409580513834953 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6074349284172058 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20540860295295715 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14683854579925537 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5525078773498535 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13807037472724915 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7101224064826965 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8391534090042114 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021332889795303345‹«èÅ:’Á‚(:‡ »…÷Ò¥¯òÝ( disk.nvme0c0n1.out‚22984.52392578125 disk.nvme0n1.in‚7.1875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 network.sent‚ 7.758375e+06 network.recv‚ 1.184256e+06% disk./.usageGB‚1167.6776657104492 disk.md0.in‚0 disk.rbd0.in‚0$ memory_percent‚96.99082374572754' proc.memory.availableMB‚ 493.0234375 proc.memory.rssMB‚2900.375' disk.nvme1c1n1.out‚16239.6416015625* proc.memory.percent‚17.702484130859375 cpu‚25.67939450371068 proc.cpu.threads‚17 disk.dm-0.in‚ 8.64453125# disk.dm-0.out‚31624.60009765625) disk./.usagePercent‚76.05781896827547 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125% disk.nvme1n1.out‚16239.6416015625 disk.nvme0c0n1.in‚7.1875& disk.nvme0n1.out‚22984.52392578125 disk.rbd1.out‚0 disk.md0.out‚ 120.7109375 disk.dm-1.in‚0! disk.dm-1.out‚7340.5693359375+[@΂(:’ »…÷ÒÖóàÞ gpu.0.gpu‚19.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.174' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.72466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O ~ŽÃ‚(:ƒ Ê…÷Òù³žÞ disk.nvme1n1.in‚ 1.45703125$ disk.nvme1n1.out‚16523.837890625 disk.rbd0.out‚0& disk.nvme1c1n1.out‚16523.837890625 proc.memory.rssMB‚2900.375* proc.memory.percent‚17.702484130859375 proc.cpu.threads‚17 network.recv‚ 1.188294e+06% disk./.usageGB‚1167.9158668518066 disk.md0.out‚ 121.34765625 disk.nvme0c0n1.in‚7.1875% disk.nvme0n1.out‚23076.9462890625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.in‚0) disk./.usagePercent‚76.07333443099212 disk.md0.in‚0! disk.dm-0.out‚31754.439453125 disk.dm-1.in‚0! disk.dm-1.out‚7586.0107421875' disk.nvme0c0n1.out‚23076.9462890625$ memory_percent‚96.99084758758545 cpu‚25.67869547633419 network.sent‚ 7.816525e+06 disk.dm-0.in‚ 8.64453125 disk.nvme0n1.in‚7.1875 disk.rbd1.out‚0( proc.memory.availableMB‚ 493.01953125Ù:fçœÄ‚(:‘ Ê…÷Ò×÷Þ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚282.275' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.09166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰M±“Å‚(:ˆ Ù…÷ÒÊÌŸÞ disk.dm-0.in‚ 8.64453125" disk.dm-0.out‚31842.2177734375 disk.dm-1.in‚0" disk.dm-1.out‚7586.23291015625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125( proc.memory.availableMB‚ 493.01953125 proc.cpu.threads‚17 network.sent‚ 7.822232e+06 disk.md0.in‚0 disk.md0.out‚ 121.63671875 disk.nvme0n1.in‚7.1875 disk.rbd0.in‚0( disk.nvme1c1n1.out‚16576.47119140625& disk.nvme1n1.out‚16576.47119140625 proc.memory.rssMB‚2900.375 network.recv‚ 1.191911e+06' disk.nvme0c0n1.out‚23112.9482421875% disk.nvme0n1.out‚23112.9482421875 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚96.99084758758545* proc.memory.percent‚17.702484130859375 cpu‚25.67789641674103) disk./.usagePercent‚76.07336375092551$ disk./.usageGB‚1167.916316986084 disk.nvme0c0n1.in‚7.1875 disk.rbd1.in‚0Œï}7žÆ‚(:“ Ù…÷Ò†ÆçÞ gpu.0.gpu‚91.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚274.822' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.60733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<øŽú2Ç- _wandbruntime‚4591  _runtime‚4591H”·\ÉÈà ' lr-AdamW/pg1‚0.00015699880916168535 ' lr-AdamW/pg2‚0.00015699880916168535 trainer/global_step‚4849 % _timestamp‚1.7845296276719892e+09 _runtime‚4591.940210001 _step‚396¿%EKÎÉÈ ' lr-AdamW/pg1‚0.00015699880916168535 ' lr-AdamW/pg2‚0.00015699880916168535 trainer/global_step‚4849 % _timestamp‚1.7845296276719892e+09 _runtime‚4591.940210001 _step‚396Œ¾ „2Ê- _wandbruntime‚4591  _runtime‚4591`Jcwæ'ËàO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47986431 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5626257297780 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47996574 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8464905 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4849 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚738129355433104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748109793526784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5635097997824 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743755970331518 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5635097997824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166990921720320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748109793526784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚742475053992448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚738129372472464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚742475053992448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚397 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39273651 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748110151990272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11608713 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39273651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161323483809280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5626597859054 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5667419298816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667322418176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚743755612730884 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47986431 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748110151990272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20073343 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39273649 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5634756573696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166990806227456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161323502421504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87260080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚742475036953088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128&¨`<+ÌåO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87270225 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743755970331518 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5635097997824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166990921720320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748109793526784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚742475053992448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚738129372472464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚742475053992448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚397 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39273651 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748110151990272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11608713 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39273651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161323483809280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5626597859054 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5667419298816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667322418176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚743755612730884 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47986431 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748110151990272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20073343 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39273649 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5634756573696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166990806227456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1613235é9ÉŠÀ 02421504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87260080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚742475036953088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47986431 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5626257297780 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47996574 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8464905 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4849 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚738129355433104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748109793526784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5635097997824 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4849 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8466642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11610748 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748259536598528 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11611028 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚742623311455232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48006244 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39281444 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47996087 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48006244 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20077387 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743904509387354 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87287688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5635866672640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39281444 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167025220240384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚738276803139936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39281442 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748259178127872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5627706247418 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚7426233114552326 øõ&ÏïM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5635866672640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39281444 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167025220240384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚738276803139936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39281442 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748259178127872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5627706247418 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚742623311455232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5636208103936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87277529 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5636208103936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 $ _timestamp‚1.784529628304949e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87287688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚738276786100576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚4592.573276984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4849 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8466642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11610748 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748259536598528 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11611028 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚742623311455232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48006244 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39281444 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47996087 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48006244 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20077387 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743904509387354 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87287688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ŽÊùûN2Ð- _wandbruntime‚4592  _runtime‚45920Ks·Ñ 1loss/SRqqt1/detection_loss‚0.6665898263790446 1loss/SRqqt2/detection_loss‚0.9110459256803449 epoch‚12 % _timestamp‚1.7845296283065188e+09 2loss/SRqqt2/assignment_loss‚0.5540482537820717 /loss/FRt2/detection_loss‚0.5066026559712439 /loss/FBt2/detection_loss‚1.0955140779167414 _runtime‚4592.574416795 _step‚399 /loss/FRt2/assignment_loss‚1.786511076331825 2loss/SRqqt1/assignment_loss‚0.5397369732303713 trainer/global_step‚4849 1loss/FBt1/assignment_loss‚0.08947937657355715 2loss/FBt2/assignment_loss‚0.016843168437886503 /loss/FRt1/detection_loss‚1.5057522842658513 /loss/FBt1/detection_loss‚0.3792598669463063 $loss/total_loss‚8.82193783635884 0loss/FRt1/assignment_loss‚0.7705543508435961yžx' Ò† 1loss/FBt1/assignment_loss‚0.08947937657355715 2loss/FBt2/assignment_loss‚0.016843168437886503 /loss/FRt1/detection_loss‚1.5057522842658513 /loss/FBt1/detection_loss‚0.3792598669463063 $loss/total_loss‚8.82193783635884 0loss/FRt1/assignment_loss‚0.7705543508435961 1loss/SRqqt1/detection_loss‚0.6665898263790446 1loss/SRqqt2/detection_loss‚0.9110459256803449 epoch‚12 % _timestamp‚1.7845296283065188e+09 2loss/SRqqt2/assignment_loss‚0.5540482537820717 /loss/FRt2/detection_loss‚0.5066026559712439 /loss/FBt2/detection_loss‚1.0955140779167414 _runtime‚4592.574416795 _step‚399 /loss/FRt2/assignment_loss‚1.786511076331825 2loss/SRqqt1/assignment_loss‚0.5397369732303713 trainer/global_step‚4849FÁìÓ‚(:… è…÷Ò¥ÛÌÞ disk.nvme0c0n1.in‚7.1875% disk.nvme0c0n1.out‚23147.83203125# disk.nvme0n1.out‚23147.83203125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.45703125( disk./.usagePercent‚76.0733443699526 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125& disk.nvme1n1.out‚16576.98388671875* proc.memory.percent‚17.702484130859375 cpu‚25.677206667414858 disk.md0.out‚ 121.90234375 disk.dm-0.in‚ 8.64453125 disk.rbd1.in‚0$ memory_percent‚96.99180126190186( proc.memory.availableMB‚ 492.86328125 network.recv‚ 1.194809e+06% disk./.usageGB‚1167.9160194396973 disk.dm-1.in‚0 disk.nvme0n1.in‚7.1875( disk.nvme1c1n1.out‚16576.98388671875 disk.rbd1.out‚0 proc.memory.rssMB‚2900.375 proc.cpu.threads‚17 network.sent‚ 7.837997e+06" disk.dm-0.out‚31876.9248046875" disk.dm-1.out‚7586.33447265625ö@4‘Ô‚(:† è…÷Ò ùŒß gpu.0.gpu‚72.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚277.89' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.63 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=¿˜—Õ‚(:Œ ÷…÷ÒåýÝ) disk./.usagePercent‚76.07339605254705 disk.dm-0.in‚ 8.64453125( disk.nvme0c0n1.out‚23159.61669921875 disk.rbd0.in‚0 disk.rbd1.out‚0 cpu‚25.677071889495725 proc.cpu.threads‚17 network.recv‚ 1.199086e+06 disk.md0.in‚0 disk.md0.out‚ 122.05859375# disk.dm-0.out‚31889.36279296875" disk.dm-1.out‚7586.53662109375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚16578.18896484375 network.sent‚ 7.887792e+06% disk./.usageGB‚1167.9168128967285& disk.nvme0n1.out‚23159.61669921875 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚16578.18896484375 proc.memory.rssMB‚2900.375* proc.memory.percent‚17.702484130859375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875$ memory_percent‚96.99339866638184' proc.memory.availableMB‚ 492.6015625{„ ©Ö‚(:’ ÷…÷Ò­þþÝ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.93' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.97666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„°­êׂ(:ô ††÷Ò©ÌäÝ disk.rbd1.out‚0 disk.dm-0.in‚ 8.64453125" disk.dm-1.out‚7586.61865234375 disk.nvme0c0n1.in‚7.1875$ memory_percent‚96.99344635009766 proc.cpu.threads‚17 disk.md0.out‚ 122.40234375 network.sent‚ 7.892679e+06) disk./.usagePercent‚75.88087267332911 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 disk.nvme1n1.in‚ 1.45703125 disk.md0.in‚0# disk.dm-0.out‚31890.09716796875 network.recv‚ 1.202511e+06% disk./.usageGB‚1164.9610977172852 disk.rbd0.in‚0 disk.nvme1n1.out‚16578.75 disk.dm-1.in‚0& disk.nvme0c0n1.out‚23160.619140625 disk.nvme0n1.in‚7.1875% proc.memory.availableMB‚ 492.59375 disk.nvme1c1n1.out‚16578.75 disk.rbd1.in‚0$ disk.nvme0n1.out‚23160.619140625 proc.memory.rssMB‚2900.375* proc.memory.percent‚17.702484130859375 cpu‚25.676449370175483f}IžØ‚(:“ ††÷Ò’¤ôÝ gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.504' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.50133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g#)2Ù- _wandbruntime‚4636  _runtime‚4636j»BÇÚÁ & lr-AdamW/pg1‚0.0001565279842045587 & lr-AdamW/pg2‚0.0001565279842045587 trainer/global_step‚4899 % _timestamp‚1.7845296719746232e+09 _runtime‚4636.242736619 _step‚400áÔìdÌÛÆ _runtime‚4636.242736619 _step‚400 & lr-AdamW/pg1‚0.0001565279842045587 & lr-AdamW/pg2‚0.0001565279842045587 trainer/global_step‚4899 % _timestamp‚1.7845296719746232e+09ƒþˆö2Ü- _wandbruntime‚4636  _runtime‚4636ê’;ý ÝáO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48471083 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845296719754596e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88134382 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5690325675520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚4636.243482751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5690667099648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745500888806704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚755579087735808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5723158306816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚755579087735808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39663301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168705783695872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5723267770368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segmeÒ‚/pênts.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11723239 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚749888762060288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5690667099648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚749888779099648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚749888779099648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39663299 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚755579446199296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11722950 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48471083 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39663299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20274847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5682081072462 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚755579446199296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20274555 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8551608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162982515925504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751182986918526 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39663301 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745500905846064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48481226 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8551605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162982497313280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚749888762060288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751182629317892 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48481226 VÐ4ì'ÞæO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88144527 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168705655620096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚749888779099648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚749888779099648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39663299 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚755579446199296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11722950 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48471083 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39663299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20274847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5682081072462 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚755579446199296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20274555 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8551608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162982515925504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751182986918526 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39663301 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745500905846064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48481226 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8551605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162982497313280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚749888762060288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751182629317892 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48481226 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48471083 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845296719754596e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88134382 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5690325675520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚4636.243482751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5690667099648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745500888806704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚755579087735808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5723158306816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚755579087735808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39663301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168705783695872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5723267770368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11723239 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚749888762060288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5690667099648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88161990 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5682848899440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48490896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745648336513536 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39671094 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20278847 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5724379180032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88151831 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20278552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8553339 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5691777205760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11725213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163015696195584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39671092 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48490896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163015677583360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750037036562432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8553342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11725505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751331525974362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 % _timestamp‚1.7845296725803196e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5724273917952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39671094 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚755728472336896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5683189460826 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88161990 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5724273917952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39671094 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚755728472336896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5683189460826 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88161990 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88161990 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5682848899440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48490896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745648336513536 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39671094 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20278847 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5724379180032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_trÌ”vain_batch_end/requested_bytes.all.peak‚ 23704151647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88151831 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20278552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8553339 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5691777205760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11725213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163015696195584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39671092 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48490896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163015677583360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750037036562432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8553342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11725505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751331525974362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344’³ì°2â- _wandbruntime‚4636  _runtime‚4636Ìv)ó ム_runtime‚4636.853408121 0loss/FRt2/assignment_loss‚1.9093624266646319 /loss/FRt2/detection_loss‚0.5349988156380158 _step‚403 %loss/total_loss‚8.482753379903333 epoch‚12 0loss/FRt1/assignment_loss‚0.8414403467558215 2loss/SRqqt1/assignment_loss‚0.5380313180950587 2loss/FBt1/assignment_loss‚0.014429119308540784 /loss/FRt1/detection_loss‚1.4154377647556473 1loss/SRqqt1/detection_loss‚0.6202401554174874 trainer/global_step‚4899 2loss/SRqqt2/assignment_loss‚0.5622225797406871 1loss/FBt2/assignment_loss‚0.02313521644092112 1loss/SRqqt2/detection_loss‚0.8691897784958925 /loss/FBt1/detection_loss‚0.5500382267270604 /loss/FBt2/detection_loss‚0.6042276318635671 % _timestamp‚1.7845296725847921e+09ETÈ„äˆ 0loss/FRt1/assignment_loss‚0.8414403467558215 2loss/SRqqt1/assignment_loss‚0.5380313180950587 2loss/FBt1/assignment_loss‚0.014429119308540784 /loss/FRt1/detection_loss‚1.4154377647556473 1loss/SRqqt1/detection_loss‚0.6202401554174874 trainer/global_step‚4899 2loss/SRqqt2/assignment_loss‚0.5622225797406871 1loss/FBt2/assignment_loss‚0.02313521644092112 1loss/SRqqt2/detection_loss‚0.8691897784958925 /loss/FBt1/detection_loss‚0.5500382267270604 /loss/FBt2/detection_loss‚0.6042276318635671 % _timestamp‚1.7845296725847921e+09 _runtime‚4636.853408121 0loss/FRt2/assignment_loss‚1.9093624266646319 /loss/FRt2/detection_loss‚0.5349988156380158 _step‚403 %loss/total_loss‚8.482753379903333 epoch‚12“—ù^lŽå‚(:ƒ •†÷Ò“º£Þ) disk./.usagePercent‚75.88086944316696% disk./.usageGB‚1164.9610481262207 disk.md0.in‚0 disk.md0.out‚ 122.71484375 disk.rbd0.out‚0( proc.memory.availableMB‚ 492.77734375* proc.memory.percent‚17.702484130859375 disk.dm-1.in‚0! disk.dm-1.out‚7586.7958984375 disk.nvme1c1n1.in‚ 1.45703125 proc.memory.rssMB‚2900.375 cpu‚25.67593012551748 proc.cpu.threads‚17 network.recv‚ 1.206134e+06! disk.dm-0.out‚31896.095703125 disk.nvme0c0n1.in‚7.1875& disk.nvme0c0n1.out‚23166.794921875 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.45703125 disk.dm-0.in‚ 8.64453125$ disk.nvme0n1.out‚23166.794921875 disk.rbd0.in‚0' disk.nvme1c1n1.out‚16579.4326171875% disk.nvme1n1.out‚16579.4326171875$ memory_percent‚96.99232578277588 network.sent‚ 7.910393e+06M…,!žæ‚(:“ •†÷ÒÍÈçÞ gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚259.766' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.58866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä´0Óžç‚(:“ ¤†÷ÒîàÞ gpu.0.gpu‚55.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.024' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.67466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ¯ˆÓ…è‚(:ú ¤†÷ÒåôïÞ$ memory_percent‚96.99385166168213* proc.memory.percent‚17.702484130859375 network.sent‚ 7.962014e+06% disk./.usageGB‚1165.2118606567383 disk.dm-0.in‚ 8.64453125! disk.dm-1.out‚7586.9833984375 disk.nvme0n1.in‚7.1875 disk.rbd1.in‚0% disk.nvme1n1.out‚16580.2666015625 cpu‚25.675733399345262! disk.dm-0.out‚31896.927734375 disk.nvme0c0n1.in‚7.1875# disk.nvme0c0n1.out‚ 23168.171875 disk.nvme1c1n1.in‚ 1.45703125( proc.memory.availableMB‚ 492.52734375 proc.memory.rssMB‚2900.375 proc.cpu.threads‚17 network.recv‚ 1.210342e+06( disk./.usagePercent‚75.8972063609665 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚16580.2666015625 disk.nvme1n1.in‚ 1.45703125 disk.dm-1.in‚0! disk.nvme0n1.out‚ 23168.171875 disk.rbd1.out‚0 disk.md0.in‚0 disk.md0.out‚ 123.28125Ö©'Á2é- _wandbruntime‚4679  _runtime‚4679ýŒFÈê ' lr-AdamW/pg1‚0.00015605284711294902 ' lr-AdamW/pg2‚0.00015605284711294902 trainer/global_step‚4949 $ _timestamp‚1.784529714922064e+09 _runtime‚4679.192417676 _step‚404¿m] ÍëÇ _step‚404 ' lr-AdamW/pg1‚0.00015605284711294902 ' lr-AdamW/pg2‚0.00015605284711294902 trainer/global_step‚4949 $ _timestamp‚1.784529714922064e+09 _runtime‚4679.192417676”J0Œ2ì- _wandbruntime‚4679  _runtime‚4679n5K½æ'íàO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48955735 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48965878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757302504206848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5779095779840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚763048740408320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5746236201472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757302504206848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8638308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11837031 $ _timestamp‚1.784529714927412e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5737564285870 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚758609645904900 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11837318 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5778994704896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚763048381944832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89018829 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170420625209344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚758610003505534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚763048740408320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20475626 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184ðÛi›ë'îåO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757302504206848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8638308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11837031 $ _timestamp‚1.784529714927412e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5737564285870 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚758609645904900 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11837318 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5778994704896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚763048381944832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89018829 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170420625209344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚758610003505534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚763048740408320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20475626 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48955735 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48965878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757302504206848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5779095779840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚763048740408320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5746236201472•>GT2ï- _wandbruntime‚4679  _runtime‚4679¶÷MFê ðëM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5747004876288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40060744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40060744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11839280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 32yù58470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11839559 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89026133 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5738672674234 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5747346307584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757450778708992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164674691087360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚758758542561370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170454927528448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8640039 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89036292 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164674709699584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚763198125016576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5747346307584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8640042 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48975548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753019869887136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845297155554183e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5747004876288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40060742 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170454795265536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89036292 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20479601 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4679.823791669 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20479319 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40060742 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚406 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360aõÙ‡ö&ñðM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚757450778708992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚763197766545920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5780104178176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89036292 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164674709699584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚763198125016576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5747346307584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8640042 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48975548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753019869887136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845297155554183e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5747004876288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40060742 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170454795265536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89036292 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20479601 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4679.823791669 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20479319 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40060742 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚406 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5747004876288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40060744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40060744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11839280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11839559 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89026133 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5738672674234 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5747346307584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757450778708992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164674691087360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚758758542561370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170454927528448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8640039 FÍýÇ ' lr-AdamW/pg1‚0.00015557342840402387 ' lr-AdamW/pg2‚0.00015557342840402387 trainer/global_step‚4999 $ _timestamp‚1.784529757540985e+09 _runtime‚4721.809150386 _step‚408˜G—L62þ- _wandbruntime‚4721  _runtime‚4721ƒqDη$ÿáO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚766037020092542 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770518034617344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚764716212274688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49450530 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20676353 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89882986 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5801805303296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5801463879168 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11951054 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172135348592128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49440387 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11951345 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166300524321280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166300542933504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40442601 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4721.809973917 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚764716212274688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚770518034617344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8725008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚409 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚770517676153856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770517676153856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5801805303296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89893131 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40442601 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8725005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172135476667904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5793047499278 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5834824270848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5834933734400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760243972593264 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚764716229314048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5801463879168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚760243955553904 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11951054 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172135348592128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49440387 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11951345 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166300524321280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166300542933504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40442601 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4721.809973917 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚764716212274688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚770518034617344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8725008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚409 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚770517676153856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770517676153856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5801805303296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89893131 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40442601 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8725005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172135476667904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5793047499278 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5834824270848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5834933734400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760243972593264 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚764716229314048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5801463879168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚760243955553904 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚766185201547632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49450043 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49460200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚760391386221376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5802573978112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚4999 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5802915409408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172169644366336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40450392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20680013 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11953274 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8726739 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8726742 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5802573978112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20680314 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚770667419225600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89900435 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784529758158765e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚770667060754944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11953572 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5794155887642 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40450394 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5802915409408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚410 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166333704591360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5835939774976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4722.427104507 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40450392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚764864486776832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49450043 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89910594 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚764864503816192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚764864503816192câ+õ&ƒïM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89900435 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4722.427104507 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40450392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚764864486776832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49450043 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89910594 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚764864503816192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚764864503816192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166333723203584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172169772434944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚766185201547632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49450043 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49460200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚760391386221376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5802573978112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚4999 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5802915409408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172169644366336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40450392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20680013 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11953274 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8726739 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8726742 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5802573978112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20680314 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚770667419225600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89900435 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784529758158765e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚770667060754944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11953572 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5794155887642 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40450394 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5802915409408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚410 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166333704591360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5835939774976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12065130 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167959537825280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777986970362880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777986970362880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚767615505966864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772129937381888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5857032980992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚773463679078916 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49925039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49935182 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5890754788864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40832249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20877139 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845298016020296e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772129954421248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚773464036679550 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5848190151412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚772129954421248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚767615488927504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40832251 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5857374405120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5890636936704 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40832249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49925039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8811705 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚777987328826368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173850174761984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90757288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5857374405120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚5049 _step‚413 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20876835 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8811708 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚5049 _step‚413 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20876835 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8811708 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12065130 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167959537825280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777986970362880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777986970362880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚767615505966864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772129937381888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5857032980992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚773463679078916 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49925039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49935182 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5890754788864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40832249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20877139 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845298016020296e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772129954421248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚773464036679550 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5848190151412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚772129954421248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚767615488927504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40832251 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5857374405120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5890636936704 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40832249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49925039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8811705 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚777987328826368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173850174761984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90757288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0¸f™2“- _wandbruntime‚4766  _runtime‚4766Hã±½v”êM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4766.497404787 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40840042 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778136713434624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚778136354963968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5891744814592 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20880849 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773612218134640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12067694 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚773612575735386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90774737 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40840042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5858484511232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚767762936634336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5858484511232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40840044 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90784896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772278211884032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90784896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90774737 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173884607755776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173884462909952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚767762919594976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772278228923392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.smal¤%¨„zl_pool.allocated‚49944852 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚772278228923392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167992736707584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5891871048192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772278211884032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49934695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 144845824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5849298539664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784529802229135e+09 _step‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5858143079936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8813439 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚5049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5858143079936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚11807‰òâõ&•ïM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4766.497404787 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40840042 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778136713434624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚778136354963968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5891744814592 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20880849 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773612218134640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12067694 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚773612575735386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90774737 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40840042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5858484511232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚767762936634336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5858484511232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40840044 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90784896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772278211884032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90784896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90774737 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173884607755776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173884462909952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚767762919594976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772278228923392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49944852 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚772278228923392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167992736707584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5891871048192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772278211884032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49934695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 144845824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5849298539664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784529802229135e+09 _step‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5858143079936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8813439 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚5049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5858143079936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ž†!ÍF2–- _wandbruntime‚4766  _runtime‚4766ȳOz —„ 2loss/SRqqt1/assignment_loss‚0.5155959002607795 0loss/SRqqt2/detection_loss‚0.909436466890232 _runtime‚4766.498063378 /loss/FRt1/assignment_loss‚0.681033996520219 2loss/FBt1/assignment_loss‚0.015616791291100487 0loss/FRt2/detection_loss‚0.49806115476666746 1loss/SRqqt1/detection_loss‚0.5631091822701902 %loss/total_loss‚8.126154878017271 trainer/global_step‚5049 % _timestamp‚1.7845298022302303e+09 0loss/FRt2/assignment_loss‚1.7716405651493496 3loss/FBt2/assignment_loss‚0.0009726689401950942 /loss/FBt1/detection_loss‚0.5142215476359316 epoch‚12 2loss/SRqqt2/assignment_loss‚0.6501554589976877 /loss/FRt1/detection_loss‚1.3626864174392397 /loss/FBt2/detection_loss‚0.6436247278556779 _step‚415 8£û˜‰ epoch‚12 2loss/SRqqt2/assignment_loss‚0.6501554589976877 /loss/FRt1/detection_loss‚1.3626864174392397 /loss/FBt2/detection_loss‚0.6436247278556779 _step‚415 2loss/SRqqt1/assignment_loss‚0.5155959002607795 0loss/SRqqt2/detection_loss‚0.909436466890232 _runtime‚4766.498063378 /loss/FRt1/assignment_loss‚0.681033996520219 2loss/FBt1/assignment_loss‚0.015616791291100487 0loss/FRt2/detection_loss‚0.49806115476666746 1loss/SRqqt1/detection_loss‚0.5631091822701902 %loss/total_loss‚8.126154878017271 trainer/global_step‚5049 % _timestamp‚1.7845298022302303e+09 0loss/FRt2/assignment_loss‚1.7716405651493496 3loss/FBt2/assignment_loss‚0.0009726689401950942 /loss/FBt1/detection_loss‚0.5142215476359316Ÿ&Ó%C–™‚(:‹ ‡÷Ò³ŸÀÞ disk.nvme0n1.in‚7.1875 network.sent‚ 8.121345e+06 network.recv‚ 1.234973e+06% disk./.usageGB‚1165.1966361999512 disk.md0.in‚0# disk.dm-0.out‚35224.01806640625 disk.nvme1c1n1.in‚ 1.45703125 disk.rbd1.out‚0" disk.dm-1.out‚7587.95751953125( disk.nvme1c1n1.out‚17016.55810546875 disk.nvme1n1.in‚ 1.45703125$ memory_percent‚97.00102806091309* proc.memory.percent‚17.702484130859375 disk.dm-1.in‚0& disk.nvme0n1.out‚26065.28271484375) disk./.usagePercent‚75.89621470118556 disk.md0.out‚ 125.75390625 disk.dm-0.in‚ 8.64453125 disk.rbd0.out‚0& disk.nvme1n1.out‚17016.55810546875' proc.memory.availableMB‚ 491.3515625 disk.nvme0c0n1.in‚7.1875( disk.nvme0c0n1.out‚26065.28271484375 disk.rbd0.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚2900.375 cpu‚25.68227653867569 proc.cpu.threads‚17÷¸&½žš‚(:“ ‡÷Òƒüˆß gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚257.506' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.83533333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀúË#‘›‚(:† œ‡÷ÒŠ½÷Ý disk.rbd1.out‚0# disk.dm-0.out‚35225.43505859375 disk.dm-1.in‚0$ memory_percent‚97.00126647949219 proc.memory.rssMB‚2900.375 cpu‚25.683223433739727 disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚17017.30517578125 network.recv‚ 1.239011e+06) disk./.usagePercent‚75.89624526348899 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875& disk.nvme1n1.out‚17017.30517578125 disk.dm-0.in‚ 8.64453125 disk.rbd1.in‚0 disk.nvme1n1.in‚ 1.45703125$ proc.memory.availableMB‚491.3125* proc.memory.percent‚17.702484130859375 proc.cpu.threads‚17 disk.rbd0.in‚0 network.sent‚ 8.172847e+06% disk./.usageGB‚1165.1971054077148( disk.nvme0c0n1.out‚26066.66943359375& disk.nvme0n1.out‚26066.66943359375 disk.md0.in‚0 disk.md0.out‚ 126.03125" disk.dm-1.out‚7588.04736328125 disk.rbd0.out‚0Ûq÷Xžœ‚(:“ œ‡÷Ò™‹ÁÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.978' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.65933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùïp‚(:„ «‡÷Ò½ÛôÝ disk.md0.in‚0 disk.md0.out‚ 126.48828125 disk.nvme0c0n1.in‚7.1875 disk.nvme0n1.in‚7.1875$ memory_percent‚97.00126647949219$ proc.memory.availableMB‚491.3125* proc.memory.percent‚17.702484130859375 network.sent‚ 8.175811e+06 disk.dm-0.out‚35234.81640625 disk.dm-1.in‚0! disk.dm-1.out‚7588.1806640625( disk.nvme0c0n1.out‚26074.57373046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125 network.recv‚ 1.242511e+06) disk./.usagePercent‚75.89627433494837& disk.nvme0n1.out‚26074.57373046875 disk.nvme1n1.in‚ 1.45703125& disk.nvme1n1.out‚17019.89111328125 proc.memory.rssMB‚2900.375 cpu‚25.684262140820483 proc.cpu.threads‚17$ disk./.usageGB‚1165.197551727295 disk.dm-0.in‚ 8.64453125( disk.nvme1c1n1.out‚17019.89111328125 disk.rbd1.in‚0 disk.rbd1.out‚0À@×Ï“ž‚(:ˆ «‡÷ÒçèÐÞ gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚264.573' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.191 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y2æ2Ÿ- _wandbruntime‚4811  _runtime‚4811˜ëF È Â _runtime‚4811.527170109 _step‚416 ' lr-AdamW/pg1‚0.00015460186957592082 ' lr-AdamW/pg2‚0.00015460186957592082 trainer/global_step‚5099 $ _timestamp‚1.784529847259042e+09·#YÉÍ¡Ç $ _timestamp‚1.784529847259042e+09 _runtime‚4811.527170109 _step‚416 ' lr-AdamW/pg1‚0.00015460186957592082 ' lr-AdamW/pg2‚0.00015460186957592082 trainer/global_step‚5099 ‰Ïê2¢- _wandbruntime‚4811  _runtime‚4811:ˆ ^æ'£àO _runtime‚4811.528142186 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5912602082816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12179322 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5912602082816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚779543679528448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50409691 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5946458806784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5904013926094 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚779543662489088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41221901 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175565010136064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50419834 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21077437 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚774987039340464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12179032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚780891053266558 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21077730 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8898405 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚780890695665924 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚779543679528448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5903673364820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚779543662489088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785456264571904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785456623035392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91641735 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41221899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169618551329280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5912943506944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845298472600513e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50419834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41221901 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50409691 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚785456623035392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169618569941504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946555687424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0êP}k¤åO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845298472600513e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50419834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41221901 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50409691 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚785456623035392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169618569941504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946555687424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4811.528142186 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5912602082816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12179322 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5912602082816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚779543679528448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50409691 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5946458806784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5904013926094 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚779543662489088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41221901 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175565010136064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50419834 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2TÝ€1077437 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚774987039340464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12179032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚780891053266558 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21077730 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8898405 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚780890695665924 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚779543679528448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5903673364820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚779543662489088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785456264571904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785456623035392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91641735 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41221899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169618551329280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5912943506944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128¡–TÆ2¥- _wandbruntime‚4812  _runtime‚4812[Cþ6ñ&¦ëM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5914053613056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785606007643648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91659198 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5947574661632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50429504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚785605649172992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚779691936991232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5905122314458 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5947675729408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785606007643648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175599425940992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚4812.156460744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91649039 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5913712181760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21081428 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚775134470007936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91659198 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5913712181760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚779691954030592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175599306260992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169651731599360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5914053613056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚775134452968576 _step‚418 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845298478881388e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8900142 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41229692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50419347 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41229694 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180˜Pªaö&§ðM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚4812.156460744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91649039 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5913712181760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21081428 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚775134470007936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91659198 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5913712181760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚779691954030592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175599306260992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169651731599360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5914053613056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚775134452968576 _step‚418 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845298478881388e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8900142 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41229692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50419347 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41229694 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5914053613056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785606007643648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91659198 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5947574661632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50429504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚785605649172992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚779691936991232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5905122314458 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5947675729408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785606007643648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175599425940992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segmeªWÒ|½%nts.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92505892 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41611551 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50904486 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5968512608768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚782358572714064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚786957387596288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚788317712252932 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚421 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171277564833280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚792925917244416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8985105 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92516037 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚788318069853566 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784529892975923e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50894343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12293620 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177279960368640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5968512608768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚782358555674704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5959497139502 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50894343 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8985108 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12293333 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5968171184640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚786957404635648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41611549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792925558780928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6002376923136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚792925917244416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41611551 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚786957387596288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21278438 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6002271653888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50904486 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21278728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171277583445504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5968171184640 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792925558780928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5149 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177279836487168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚788317712252932 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚421 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171277564833280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚792925917244416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8985105 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92516037 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚788318069853566 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784529892975923e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50894343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12293620 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177279960368640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5968512608768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚782358555674704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5959497139502 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50894343 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8985108 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12293333 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5968171184640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚786957404635648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41611549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792925558780928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6002376923136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚792925917244416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41611551 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚786957387596288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21278438 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6002271653888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50904486 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21278728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171277583445504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5968171184640 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792925558780928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5149 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177279836487168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92505892 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41611551 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50904486 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5968512608768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚782358572714064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚786957387596288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0¥£5¶2·- _wandbruntime‚4857  _runtime‚4857&«`Xñ&¸ëM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177314135210496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787105662098432 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5969281283584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41619344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚782506003381536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92523341 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793074943382016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12295911 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6003390107136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5969622714880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚782505986342176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92533500 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171310763715584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787105679137792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50903999 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787105662098432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8986839 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5960264966480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92523341 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50914156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177314256987648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6003493272064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92533500 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21282753 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41619344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5960605527866 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21282462 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793075301852672®¸- ¹ðM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171310763715584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787105679137792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50903999 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787105662098432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8986839 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5960264966480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92523341 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50914156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177314256987648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6003493272064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92533500 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21282753 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41619344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5960605527866 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21282462 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793075301852672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177314135210496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787105662098432 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5969281283584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41619344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚782506003381536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92523341 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793074943382016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12295911 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6003390107136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5969622714880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚782505986342176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92533500 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647¦!ÁBÈ2º- _wandbruntime‚4857  _runtime‚4857NÅÁ »ƒ 0loss/FRt1/assignment_loss‚0.7028729165422505 1loss/SRqqt1/assignment_loss‚0.538544534125398 2loss/FBt2/assignment_loss‚0.018224633310613365 /loss/FRt2/detection_loss‚0.5327875499026945 1loss/SRqqt2/detection_loss‚0.9276495176539021 /loss/FBt1/detection_loss‚0.5963832380497683 /loss/FBt2/detection_loss‚0.6047687909033682 %loss/total_loss‚8.311621835171165 2loss/FBt1/assignment_loss‚0.007447578686683643 /loss/FRt1/detection_loss‚1.4608321466968908 _runtime‚4857.876430906 2loss/SRqqt2/assignment_loss‚0.4960170658241862 1loss/SRqqt1/detection_loss‚0.6013678968083491 trainer/global_step‚5149 0loss/FRt2/assignment_loss‚1.8247259666670614 epoch‚12 % _timestamp‚1.7845298936085823e+09 _step‚423 Ù¼ˆ % _timestamp‚1.7845298936085823e+09 _step‚423 0loss/FRt1/assignment_loss‚0.7028729165422505 1loss/SRqqt1/assignment_loss‚0.538544534125398 2loss/FBt2/assignment_loss‚0.018224633310613365 /loss/FRt2/detection_loss‚0.5327875499026945 1loss/SRqqt2/detection_loss‚0.9276495176539021 /loss/FBt1/detection_loss‚0.5963832380497683 /loss/FBt2/detection_loss‚0.6047687909033682 %loss/total_loss‚8.311621835171165 2loss/FBt1/assignment_loss‚0.007447578686683643 /loss/FRt1/detection_loss‚1.4608321466968908 _runtime‚4857.876430906 2loss/SRqqt2/assignment_loss‚0.4960170658241862 1loss/SRqqt1/detection_loss‚0.6013678968083491 trainer/global_step‚5149 0loss/FRt2/assignment_loss‚1.8247259666670614 epoch‚12§œ¢C$‘½‚(:† ç‡÷ÒƼ˜Þ proc.cpu.threads‚17) disk./.usagePercent‚76.07311403454369 disk.dm-0.in‚ 8.64453125 disk.dm-1.in‚0( disk.nvme0c0n1.out‚28567.68994140625 disk.rbd0.in‚0 disk.rbd1.out‚0% disk.nvme1n1.out‚17318.3837890625 network.sent‚ 8.253019e+06 disk.md0.out‚ 127.7734375! disk.dm-1.out‚7588.5009765625 disk.nvme0c0n1.in‚7.1875& disk.nvme0n1.out‚28567.68994140625' disk.nvme1c1n1.out‚17318.3837890625 disk.rbd1.in‚0 proc.memory.rssMB‚2900.375 network.recv‚ 1.257405e+06# disk.dm-0.out‚38023.30712890625 disk.nvme0n1.in‚7.1875 disk.nvme1n1.in‚ 1.45703125& proc.memory.availableMB‚ 491.265625$ disk./.usageGB‚1167.912483215332 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125$ memory_percent‚97.00155258178711* proc.memory.percent‚17.702484130859375 cpu‚25.688759208934805$ ož¾‚(:“ ç‡÷ÒÆß gpu.0.gpu‚94.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.812' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÓÇž¿‚(:“ ö‡÷ÒñÑéÝ disk.nvme1n1.in‚ 1.45703125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7.21484375 network.recv‚ 1.261566e+06 disk.md0.out‚ 129.90234375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2900.37890625* proc.memory.percent‚17.702507972717285 disk.dm-0.in‚8.671875 disk.nvme0c0n1.in‚ 7.21484375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 1.45703125( disk.nvme1c1n1.out‚17323.65869140625& disk.nvme1n1.out‚17323.65869140625& proc.memory.availableMB‚ 491.265625 cpu‚25.689462121567175 network.sent‚ 8.316793e+06% disk./.usageGB‚1168.9155158996582! disk.dm-0.out‚38085.119140625! disk.dm-1.out‚7588.8935546875( disk.nvme0c0n1.out‚28628.94677734375 disk.rbd0.in‚0$ memory_percent‚97.00155258178711) disk./.usagePercent‚76.13844754272321 disk.md0.in‚0& disk.nvme0n1.out‚28628.94677734375 disk.rbd1.in‚0 disk.rbd1.out‚0Ë(ÕžÀ‚(:“ ö‡÷ÒÒêß gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.876' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.95866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Ú¥É¦Á‚(:› …ˆ÷Òô’úÝ disk.md0.out‚ 130.41015625 network.recv‚ 1.264871e+06 disk.rbd0.out‚0& disk.nvme1n1.out‚17327.24560546875 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.04296875* proc.memory.percent‚17.702507972717285 proc.cpu.threads‚17 network.sent‚ 8.322521e+06 disk.rbd0.in‚0( disk.nvme1c1n1.out‚17327.24560546875 disk.rbd1.in‚0 disk.md0.in‚0# disk.dm-0.out‚39403.87548828125" disk.dm-1.out‚7589.42138671875( disk.nvme0c0n1.out‚29945.73095703125) disk./.usagePercent‚76.16001806863257% disk./.usageGB‚1169.2466773986816 disk.nvme1c1n1.in‚ 1.8203125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 1.8203125 disk.dm-0.in‚ 11.86328125& disk.nvme0n1.out‚29945.73095703125# proc.memory.rssMB‚ 2900.37890625 disk.nvme0n1.in‚ 10.04296875$ memory_percent‚97.00157642364502( proc.memory.availableMB‚ 491.26171875 cpu‚25.688677638819616 bý,žÂ‚(:“ …ˆ÷Ò—§ÀÞ gpu.0.gpu‚48.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.478' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.49266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÃ~&2Ã- _wandbruntime‚4902  _runtime‚4902…„aÚÇÄÁ & lr-AdamW/pg1‚0.0001536135573218738 & lr-AdamW/pg2‚0.0001536135573218738 trainer/global_step‚5199 % _timestamp‚1.7845299380988655e+09 _runtime‚4902.367523548 _step‚424DŽa­ÌÅÆ % _timestamp‚1.7845299380988655e+09 _runtime‚4902.367523548 _step‚424 & lr-AdamW/pg1‚0.0001536135573218738 & lr-AdamW/pg2‚0.0001536135573218738 trainer/global_step‚5199¨FZÕù2Æ- _wandbruntime‚4902  _runtime‚4902¨(øç'ÇáO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795744728839940 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178994844845056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6058142626304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795745086440574 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12408518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172936578337280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6014639791636 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51389138 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21480326 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42001201 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚794371129742848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172936596949504 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93390339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12408240 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794371112703488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42001201 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6024081710592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9071808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21480045 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93380194 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789730106087664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚794371129742848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚800394852989952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚800395211453440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚800394852989952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚794371112703488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845299380996492e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42001199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51389138 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚4902.367900568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9071805 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6023740286464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚425 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51378995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6024081710592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6014980352910 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚5199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6023740286464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0#!›Ö#ÈæO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12408518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172936578337280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6014639791636 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51389138 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21480326 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42001201 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚794371129742848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172936596949504 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93390339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12408240 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794371112703488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42001201 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6024081710592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9071808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21480045 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93380194 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789730106087664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚794371129742848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚800394852989952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚800395211453440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚800394852989952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚794371112703488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845299380996492e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42001199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51389138 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚4902.367900568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9071805 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6023740286464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚425 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51378995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6024081710592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ^LDeviceStatsMonitor.on_trainÛ"_batch_start/requested_bytes.small_pool.allocated‚ 6014980352910 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚5199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6023740286464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795744728839940 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178994844845056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6058142626304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795745086440574©sý8P2É- _wandbruntime‚4903  _runtime‚4903|ïèñ&ÊëM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93407802 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42008992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6016088741274 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4903.000367651 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚800544596061696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179029137556992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42008994 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21484066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179029015779840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚794519404244992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9073539 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21484367 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6024850385408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172969758607360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6015748179888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6025191816704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172969777219584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528À·X^ö&ËðM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6059360337408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚800544596061696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚794519404244992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6059257172480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4903.000367651 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚800544596061696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179029137556992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42008994 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21484066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179029015779840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚794519404244992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9073539 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21484367 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6024850385408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172969758607360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6015748179888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6025191816704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172969777219584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93407802 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42008992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6016088741274 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 P žÒ‚(:“ £ˆ÷Òý•ºÞ gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.341' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.78033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0økóL2Ó- _wandbruntime‚4920  _runtime‚4920ÓÎ\'Ô¬{ 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6996212601661682 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2650967240333557 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3256015181541443 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8972362279891968 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27926942706108093 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20073847472667694 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31161829829216003 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7542397379875183 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6599385738372803 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 *jet/accuracy_1_of_3‚0.9194604754447937 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7548196315765381 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.15724651515483856 jet/accuracy_4_of_5‚0 epoch‚12 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7021695375442505 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6874811053276062 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7513074278831482 .particle/accuracy_4_of_4‚0.952686607837677 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26692286133766174 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22584407031536102 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044811807572841644 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 particle/accuracy_1_of_1‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.612511396408081 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18963967263698578 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7597375512123108 +jet/accuracy_3_of_3‚0.07559972256422043 )loss/val_total_loss‚8.261495557821812 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959499835968018 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +particle/sensitivity‚0.6385266184806824 particle/accuracy_2_of_5‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.862887442111969 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019266683608293533 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0427069254219532 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8038413524627686 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7851930260658264 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6796962022781372 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3110542595386505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08228463679552078 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20172517001628876 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4920136630535126 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39055824279785156 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6316519975662231 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4894142746925354 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21177928149700165 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03297287970781326 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0367407351732254 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8465104103088379 particle/accuracy_5_of_5‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8759873509407043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09889565408229828 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21124882996082306 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6896764039993286 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8763984441757202 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4519994258880615 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9106154441833496 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0199870727956295 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8382842540740967 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09476342052221298 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6682654619216919 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014319432899355888 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899393916130066 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5390938520431519 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2598506510257721 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6645197868347168 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7610487937927246 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9490519165992737 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2138519585132599 :"Purity/1FRt0SRqqt0FBt/event_pƒ*\l%!urity‚0.26836341619491577 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3469242453575134 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39412662386894226 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4787602722644806 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26836341619491577 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18490393459796906 7validation_average_jet_accuracy‚0.38731420040130615 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5651041269302368 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2503166198730469 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961074590682983 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6645197868347168 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2791679799556732 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6235888004302979 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5199999809265137 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6449467539787292 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 _runtime‚4920.063511597 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6460257172584534 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981386065483093 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996802806854248 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.661321759223938 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8763984441757202 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3506269156932831 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965754151344299 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01873329095542431 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8559901714324951 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6177613139152527 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 &particle/f_score‚0.649764895439148 trainer/global_step‚5199 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7404780387878418 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5462918281555176 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07085315138101578 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16742776334285736 *jet/accuracy_1_of_4‚0.9756873846054077 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7497491836547852 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27813059091567993 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21451500058174133 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7676882147789001 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8465104103088379 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7617717385292053 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8028429746627808 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6106985211372375 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47513994574546814 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7106438875198364 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5627491474151611 _step‚428 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2179480791091919 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7523066401481628 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7384768128395081 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7845299557947776e+09 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8865893483161926 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7834543585777283 particle/accuracy_1_of_3‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6641815900802612 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4365138113498688 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972619414329529 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11189772933721542 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014319432899355888 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.605962872505188 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7999810576438904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.020817402750253677 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 +jet/accuracy_3_of_4‚0.08947824686765671 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22303526103496552 (particle/accuracy‚0.8521903157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7105292081832886 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08283822983503342 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6725926399230957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7340196371078491 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7069365978240967 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987666010856628 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505ˆÎê·=Õ±{ =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7834543585777283 particle/accuracy_1_of_3‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6641815900802612 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4365138113498688 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972619414329529 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11189772933721542 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.014319432899355888 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.605962872505188 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7999810576438904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.020817402750253677 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 +jet/accuracy_3_of_4‚0.08947824686765671 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22303526103496552 (particle/accuracy‚0.8521903157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7105292081832886 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08283822983503342 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6725926399230957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7340196371078491 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7069365978240967 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987666010856628 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6996212601661682 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2650967240333557 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3256015181541443 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8972362279891968 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27926942706108093 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20073847472667694 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31161829829216003 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7542397379875183 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6599385738372803 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 *jet/accuracy_1_of_3‚0.9194604754447937 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7548196315765381 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.15724651515483856 jet/accuracy_4_of_5‚0 epoch‚12 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7021695375442505 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6874811053276062 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7513074278831482 .particle/accuracy_4_of_4‚0.952686607837677 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26692286133766174 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22584407031536102 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044811807572841644 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 particle/accuracy_1_of_1‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.612511396408081 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18963967263698578 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7597375512123108 +jet/accuracy_3_of_3‚0.07559972256422043 )loss/val_total_loss‚8.261495557821812 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959499835968018 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +particle/sensitivity‚0.6385266184806824 particle/accuracy_2_of_5‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.862887442111969 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019266683608293533 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0427069254219532 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8038413524627686 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7851930260658264 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6796962022781372 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3110542595386505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08228463679552078 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20172517001628876 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4920136630535126 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39055824279785156 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6316519975662231 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4894142746925354 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21177928149700165 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03297287970781326 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0367407351732254 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8465104103088379 particle/accuracy_5_of_5‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8759873509407043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09889565408229828 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21124882996082306 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6896764039993286 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8763984441757202 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4519994258880615 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9106154441833496 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0199870727956295 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8382842540740967 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09476342052221298 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6682654619216919 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.014319432899355888 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899393916130066 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5390938520431519 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2598506510257721 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6645197868347168 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7610487937927246 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9490519165992737 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2138519585132599 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26836341619491577 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3469242453575134 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39412662386894226 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4787602722644806 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26836341619491577 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18490393459796906 7validation_average_jet_accuracy‚0.38731420040130615 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5651041269302368 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2503166198730469 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961074590682983 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6645197868347168 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2791679799556732 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6235888004302979 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5199999809265137 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6449467539787292 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 _runtime‚4920.063511597 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6460257172584534 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981386065483093 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996802806854248 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.661321759223938 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8763984441757202 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3506269156932831 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965754151344299 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01873329095542431 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8559901714324951 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6177613139152527 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 &particle/f_score‚0.649764895439148 trainer/global_step‚5199 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7404780387878418 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5462918281555176 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07085315138101578 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16742776334285736 *jet/accuracy_1_of_4‚0.9756873846054077 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7497491836547852 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27813059091567993 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21451500058174133 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7676882147789001 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8465104103088379 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7617717385292053 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8028429746627808 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6106985211372375 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47513994574546814 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7106438875198364 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5627491474151611 _step‚428 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2179480791091919 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7523066401481628 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7384768128395081 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7845299557947776e+09 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8865893483161926 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625¬™,Œ“Ö‚(:ˆ ²ˆ÷Ò†çÔÞ network.recv‚ 1.275571e+06% disk./.usageGB‚1166.5289344787598# disk.dm-0.out‚39415.08935546875 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.04296875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125) proc.memory.percent‚17.70310401916504 network.sent‚ 8.406779e+06 disk.md0.in‚0 disk.dm-0.in‚13.125( disk.nvme0c0n1.out‚29956.95458984375& disk.nvme0n1.out‚29956.95458984375 disk.rbd1.in‚0$ memory_percent‚97.00164794921875 cpu‚25.695538665383474) disk./.usagePercent‚75.98299524369058 disk.md0.out‚ 132.0546875 disk.nvme0n1.in‚ 10.04296875& disk.nvme1c1n1.out‚17334.259765625 disk.rbd1.out‚0 disk.nvme1n1.in‚ 3.08203125" proc.memory.availableMB‚491.25 disk.dm-1.out‚ 7592.97265625 disk.rbd0.in‚0$ disk.nvme1n1.out‚17334.259765625" proc.memory.rssMB‚ 2900.4765625 proc.cpu.threads‚17p·¡ež×‚(:“ ²ˆ÷Òëü¢ß gpu.0.gpu‚42.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚289.733' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.57766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x[i(“Ø‚(:ˆ Áˆ÷Òæ£Þ disk.rbd0.in‚0& disk.nvme0c0n1.out‚29961.212890625 disk.nvme0n1.in‚ 10.04296875$ memory_percent‚97.00164794921875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 3.08203125& disk.nvme1n1.out‚17335.22216796875 disk.md0.out‚ 132.34375# disk.dm-0.out‚39419.51123046875! disk.dm-1.out‚7593.1611328125$ disk.nvme0n1.out‚29961.212890625" proc.memory.rssMB‚ 2900.4765625) disk./.usagePercent‚75.98299996469682 disk.rbd0.out‚0( disk.nvme1c1n1.out‚17335.22216796875 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.in‚0! disk.nvme0c0n1.in‚ 10.04296875" proc.memory.availableMB‚491.25 network.sent‚ 8.451913e+06 network.recv‚ 1.279853e+06% disk./.usageGB‚1166.5290069580078 disk.nvme1c1n1.in‚ 3.08203125 disk.dm-0.in‚13.125 disk.dm-1.in‚0) proc.memory.percent‚17.70310401916504 cpu‚25.694752851087483ÚàÙ‚(:’ Áˆ÷Ò•èÓÞ gpu.0.gpu‚49.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.54' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.18000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÇY]2Ú- _wandbruntime‚4964  _runtime‚4964ƒ­øÆÛÀ _step‚429 & lr-AdamW/pg1‚0.0001531131978393062 & lr-AdamW/pg2‚0.0001531131978393062 trainer/global_step‚5249 $ _timestamp‚1.784529999970451e+09 _runtime‚4964.239559872q”@"ËÜÅ trainer/global_step‚5249 $ _timestamp‚1.784529999970451e+09 _runtime‚4964.239559872 _step‚429 & lr-AdamW/pg1‚0.0001531131978393062 & lr-AdamW/pg2‚0.0001531131978393062­ñ3G2Ý- _wandbruntime‚4964  _runtime‚4964 ¯°ÞßO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174618189115904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚809826808910336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180752906233856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚5249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42513657 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6099627224064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6099968648192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12559564 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚805113662884703 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚805113305284069 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚809826808910336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42513657 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _runtime‚4964.239661663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9162624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6099968648192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42513659 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 609’@§d9627224064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 # _timestamp‚1.78452999997118e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94468943 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803727198725632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21722185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799022895469488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51955284 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803727181686272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51955284Ò±/ê'ßäO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94458798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚809827167373824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12559864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6090426853941 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803727198725632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51955284 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174618189115904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚809826808910336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180752906233856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚5249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42513657 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6099627224064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6099968648192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12559564 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚805113662884703 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚805113305284069 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚809826808910336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42513657 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _runtime‚4964.239661663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9162624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6099968648192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42513659 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6099627224064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 # _timestamp‚1.78452999997118e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94468943 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803727198725632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21722185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799022895469488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51955284 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803727181686272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚809976193511424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚803875473227776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6101078754304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6091875803579 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42521452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174651369385984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94486406 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42521450 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21726210 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799170309097600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6135856371200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚805261844339793 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51954797 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚803875473227776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94486406 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6101078754304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9164355 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180787207144960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚803875456188416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚431 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94476247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚809976551982080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42521452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180787322630656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452…QñmpâìM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚803875456188416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚431 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94476247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚809976551982080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42521452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180787322630656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51954797 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42521450 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚809976193511424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚803875473227776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6101078754304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6091875803579 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42521452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174651369385984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94486406 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42521450 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21726210 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 ZFDeviceStatsMonitor.on_train_batch_end/reqT™d‚ uested_bytes.large_pool.freed‚799170309097600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6135856371200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚805261844339793 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51954797 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚803875473227776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94486406 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6101078754304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9164355 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180787207144960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¯ÁçW2ã- _wandbruntime‚4964  _runtime‚4964l¤a ä„ 2loss/SRqqt1/assignment_loss‚0.5795159334729426 3loss/FBt2/assignment_loss‚0.0013120932538806694 1loss/SRqqt2/detection_loss‚0.9047297818320138 0loss/FBt1/detection_loss‚0.44033800156746383 2loss/FBt1/assignment_loss‚0.008831506434188483 .loss/FRt1/detection_loss‚1.462351749880414 /loss/FBt2/detection_loss‚0.6290137885717323 0loss/FRt1/assignment_loss‚0.7247688387214228 /loss/FRt2/assignment_loss‚1.804864729410462 1loss/SRqqt1/detection_loss‚0.6169800001316925 epoch‚13 trainer/global_step‚5249 2loss/SRqqt2/assignment_loss‚0.4719830814690535 /loss/FRt2/detection_loss‚0.5067407192126605 %loss/total_loss‚8.151430223957927 % _timestamp‚1.7845300005999877e+09 _runtime‚4964.867762272 _step‚432#­ºå‰ 2loss/FBt1/assignment_loss‚0.008831506434188483 .loss/FRt1/detection_loss‚1.462351749880414 /loss/FBt2/detection_loss‚0.6290137885717323 0loss/FRt1/assignment_loss‚0.7247688387214228 /loss/FRt2/assignment_loss‚1.804864729410462 1loss/SRqqt1/detection_loss‚0.6169800001316925 epoch‚13 trainer/global_step‚5249 2loss/SRqqt2/assignment_loss‚0.4719830814690535 /loss/FRt2/detection_loss‚0.5067407192126605 %loss/total_loss‚8.151430223957927 % _timestamp‚1.7845300005999877e+09 _runtime‚4964.867762272 _step‚432 2loss/SRqqt1/assignment_loss‚0.5795159334729426 3loss/FBt2/assignment_loss‚0.0013120932538806694 1loss/SRqqt2/detection_loss‚0.9047297818320138 0loss/FBt1/detection_loss‚0.44033800156746383°;ß“æ‚(:ˆ Ј÷ÒÜ®Þ! disk.nvme0c0n1.in‚ 10.04296875( disk.nvme0c0n1.out‚29962.00634765625# disk.nvme1c1n1.out‚ 17335.671875 disk.nvme1n1.in‚ 3.08203125) disk./.usagePercent‚75.98296343901708# disk.dm-0.out‚39420.04638671875 disk.rbd0.out‚0 disk.rbd1.in‚0! disk.nvme1n1.out‚ 17335.671875& proc.memory.availableMB‚ 490.921875) proc.memory.percent‚17.70310401916504 cpu‚25.693964231561527 disk.md0.in‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0 proc.cpu.threads‚17 network.sent‚ 8.467723e+06 network.recv‚ 1.283214e+06 disk.dm-0.in‚13.125 disk.nvme0n1.in‚ 10.04296875& disk.nvme0n1.out‚29962.00634765625 disk.rbd0.in‚0# memory_percent‚97.0036506652832" proc.memory.rssMB‚ 2900.4765625% disk./.usageGB‚1166.5284461975098 disk.md0.out‚ 132.64453125 disk.dm-1.in‚0! disk.dm-1.out‚7593.2119140625³Åânžç‚(:“ Ј÷ÒäÏøÞ gpu.0.gpu‚67.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.54600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03r¤°‹è‚(:€ ߈÷ÒûŽøÝ disk.rbd0.out‚0 disk.rbd1.out‚0 disk.md0.in‚0 disk.md0.out‚ 132.8046875 disk.dm-0.in‚13.125 disk.dm-1.in‚0" disk.nvme0c0n1.out‚ 29967.71875 disk.nvme0n1.out‚ 29967.71875 disk.rbd1.in‚0$ memory_percent‚97.00217247009277) proc.memory.percent‚17.70310401916504% disk./.usageGB‚1166.5284271240234 disk.nvme0n1.in‚ 10.04296875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 3.08203125$ disk.nvme1n1.out‚17337.611328125' proc.memory.availableMB‚ 491.1640625" proc.memory.rssMB‚ 2900.4765625 cpu‚25.69328497643524 network.sent‚ 8.51916e+06 network.recv‚ 1.287292e+06" disk.dm-0.out‚39427.0537109375! disk.dm-1.out‚7593.4931640625 disk.nvme1c1n1.in‚ 3.08203125& disk.nvme1c1n1.out‚17337.611328125 proc.cpu.threads‚17) disk./.usagePercent‚75.98296219664702! disk.nvme0c0n1.in‚ 10.04296875£¶mDé‚(:’ ߈÷Ò•£ÂÞ gpu.0.gpu‚30.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚287.123' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.70766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÍʘê‚(: îˆ÷Ò°ËãÝ' disk.nvme0c0n1.out‚29968.9404296875 disk.rbd0.out‚0' disk.nvme1c1n1.out‚17338.2236328125 disk.rbd1.in‚0 disk.rbd1.out‚0! disk.dm-1.out‚7593.5439453125! disk.nvme0c0n1.in‚ 10.04296875$ memory_percent‚97.00217247009277' proc.memory.availableMB‚ 491.1640625 network.recv‚ 1.290725e+06 disk.nvme1c1n1.in‚ 3.08203125 disk.nvme1n1.in‚ 3.08203125% disk.nvme1n1.out‚17338.2236328125" proc.memory.rssMB‚ 2900.4765625 cpu‚25.692558494448658 proc.cpu.threads‚17 network.sent‚ 8.522124e+06 disk.nvme0n1.in‚ 10.04296875 disk.md0.in‚0 disk.md0.out‚ 133.2734375) proc.memory.percent‚17.70310401916504( disk./.usagePercent‚75.9829743718736% disk./.usageGB‚1166.5286140441895% disk.nvme0n1.out‚29968.9404296875 disk.rbd0.in‚0 disk.dm-0.in‚13.125" disk.dm-0.out‚39427.8310546875 disk.dm-1.in‚0;,!Që‚(:’ îˆ÷Ò‰ÂÞ gpu.0.gpu‚21.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚281.803' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.93433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šg(P2ì- _wandbruntime‚5009  _runtime‚50098ʽÇÆíÀ & lr-AdamW/pg2‚0.0001526087455476028 trainer/global_step‚5299 $ _timestamp‚1.784530044890783e+09 _runtime‚5009.158891393 _step‚433 & lr-AdamW/pg1‚0.0001526087455476028ÆAiËîÅ & lr-AdamW/pg1‚0.0001526087455476028 & lr-AdamW/pg2‚0.0001526087455476028 trainer/global_step‚5299 $ _timestamp‚1.784530044890783e+09 _runtime‚5009.158891393 _step‚433±þ±2ï- _wandbruntime‚5009  _runtime‚5009™ÙV‹ç'ðáO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182467927506944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚434 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52429793 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6155537750016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21924390 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6190619617792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806394428843088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚5299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811140923832832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚806394411803728 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42903307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6155537750016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12674780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚812540679471711 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811140923832832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811140906793472 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚817296103119360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95333100 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811140906793472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6155196325888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528? ì'ñæO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚5299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811140923832832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚806394411803728 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42903307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6155537750016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12674780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚812540679471711 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811140923832832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811140906793472 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚817296103119360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95333100 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811140906793472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6155196325888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182467927506944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚434 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52429793 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6155537750016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21924390 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6190619617792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806394428843088²8Bü2ò- _wandbruntime‚5009  _runtime‚5009‡‹v-óêM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚817445846191104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚806541859510560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚435 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚817445487720448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21928167 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚817445846191104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95350549 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5009.789475094 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52439449 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9251058 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42911100 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6191839892992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6147359016987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6147018455601 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812688860926801 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚812689218527547 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42911100 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6156647856128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52439449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12677112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42911102 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52449606 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176310382889984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95360708 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812688860926801 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚812689218527547 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42911100 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6156647856128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52439449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12677112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42911102 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52449606 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176310382889984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95360708 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21928167 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚817445846191104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95350549 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5009.789475094 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52439449 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9251058 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42911100 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6191839892992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6147359016987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6147018455601³úù(æ2õ- _wandbruntime‚5009  _runtime‚5009çÀGö‚ 0loss/FRt1/assignment_loss‚0.7892123703075492 0loss/FRt2/detection_loss‚0.49316547364472674 _runtime‚5009.790193125 1loss/FBt1/assignment_loss‚0.03502104473647963 2loss/FBt2/assignment_loss‚0.006576979422172344 1loss/SRqqt2/detection_loss‚0.9698846331291091 /loss/FBt1/detection_loss‚0.5126361957458329 epoch‚13 _step‚436 /loss/FRt2/assignment_loss‚1.819595283128806 2loss/SRqqt1/assignment_loss‚0.4960889029801135 /loss/FBt2/detection_loss‚0.6591705697391069 %loss/total_loss‚8.330071737264952 trainer/global_step‚5299 % _timestamp‚1.7845300455222409e+09 2loss/SRqqt2/assignment_loss‚0.4796360970506008 .loss/FRt1/detection_loss‚1.454774519076328 1loss/SRqqt1/detection_loss‚0.6143096683041261ßûÁ\ ÷‡ 1loss/FBt1/assignment_loss‚0.03502104473647963 2loss/FBt2/assignment_loss‚0.006576979422172344 1loss/SRqqt2/detection_loss‚0.9698846331291091 /loss/FBt1/detection_loss‚0.5126361957458329 epoch‚13 _step‚436 /loss/FRt2/assignment_loss‚1.819595283128806 2loss/SRqqt1/assignment_loss‚0.4960889029801135 /loss/FBt2/detection_loss‚0.6591705697391069 %loss/total_loss‚8.330071737264952 trainer/global_step‚5299 % _timestamp‚1.7845300455222409e+09 2loss/SRqqt2/assignment_loss‚0.4796360970506008 .loss/FRt1/detection_loss‚1.454774519076328 1loss/SRqqt1/detection_loss‚0.6143096683041261 0loss/FRt1/assignment_loss‚0.7892123703075492 0loss/FRt2/detection_loss‚0.49316547364472674 _runtime‚5009.790193125´´†–‰žø‚(:“ ýˆ÷Ò­öáÝ disk.dm-0.in‚13.125# disk.dm-0.out‚39429.93115234375! disk.dm-1.out‚7593.6435546875& disk.nvme0n1.out‚29970.93505859375$ memory_percent‚97.00524806976318( proc.memory.availableMB‚ 490.66015625) disk./.usagePercent‚76.04840304559963% disk.nvme1n1.out‚17339.1474609375 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.04296875 disk.nvme0n1.in‚ 10.04296875' disk.nvme1c1n1.out‚17339.1474609375" proc.memory.rssMB‚ 2900.4765625) proc.memory.percent‚17.70310401916504 network.sent‚ 8.528135e+06% disk./.usageGB‚1167.5331077575684 disk.nvme1n1.in‚ 3.08203125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0 cpu‚25.691884512879547 network.recv‚ 1.294348e+06 disk.md0.in‚0( disk.nvme0c0n1.out‚29970.93505859375 disk.rbd0.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.md0.out‚ 133.60546875¤+zçžù‚(:“ ýˆ÷ÒËÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.007' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.33566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00l¯‘ú‚(:† Œ‰÷Ò¯ñ¡Þ disk.md0.in‚0 disk.md0.out‚ 134.02734375 disk.rbd0.out‚0# disk.nvme1c1n1.out‚ 17339.828125) proc.memory.percent‚17.70310401916504 cpu‚25.691109916764383 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.dm-1.out‚7593.7451171875 disk.nvme0n1.in‚ 10.04296875 disk.nvme1c1n1.in‚ 3.08203125 disk.nvme1n1.in‚ 3.08203125! disk.nvme1n1.out‚ 17339.828125$ memory_percent‚97.00241088867188 network.recv‚ 1.297976e+06! disk.nvme0c0n1.in‚ 10.04296875( disk.nvme0c0n1.out‚31885.45361328125& disk.nvme0n1.out‚31885.45361328125 disk.rbd0.in‚0 disk.rbd1.out‚0# proc.memory.availableMB‚491.125" proc.memory.rssMB‚ 2900.4765625 network.sent‚ 8.588618e+06) disk./.usagePercent‚76.16288943431253% disk./.usageGB‚1169.2907600402832 disk.dm-0.in‚13.125# disk.dm-0.out‚41344.11279296875 disk.rbd1.in‚0ÙErû‚(:’ Œ‰÷ÒŒÁôÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.76' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.25333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àh.ðˆü‚(:ý ›‰÷Ò¾Þ§Þ# disk.nvme1n1.out‚17340.34765625 proc.cpu.threads‚17$ disk./.usageGB‚1169.290901184082 disk.nvme1c1n1.in‚ 3.08203125 disk.nvme1n1.in‚ 3.08203125 disk.dm-0.in‚13.125! disk.nvme0c0n1.in‚ 10.04296875% disk.nvme0c0n1.out‚31888.74609375 disk.nvme0n1.in‚ 10.04296875# disk.nvme0n1.out‚31888.74609375 disk.md0.out‚ 134.171875 network.sent‚ 8.595455e+06 network.recv‚ 1.301575e+06 disk.rbd1.in‚0" proc.memory.rssMB‚ 2900.4765625) proc.memory.percent‚17.70310401916504% disk.nvme1c1n1.out‚17340.34765625! disk.dm-0.out‚41347.552734375 disk.dm-1.in‚0 disk.dm-1.out‚7593.794921875$ memory_percent‚97.00241088867188# proc.memory.availableMB‚491.125 cpu‚25.69044357166007 disk.md0.in‚0) disk./.usagePercent‚76.16289862785098 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0 €ãœý‚(:‘ ›‰÷Ò¬¦ôÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚165.69' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚55.230000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àêâ52þ- _wandbruntime‚5054  _runtime‚5054–ñ¤ÉÆÿÀ % _timestamp‚1.7845300900421498e+09 _runtime‚ 5054.31033439 _step‚437 & lr-AdamW/pg1‚0.0001521002328467916 & lr-AdamW/pg2‚0.0001521002328467916 trainer/global_step‚5349ÛÒ2$ˀŠtrainer/global_step‚5349 % _timestamp‚1.7845300900421498e+09 _runtime‚ 5054.31033439 _step‚437 & lr-AdamW/pg1‚0.0001521002328467916 & lr-AdamW/pg2‚0.0001521002328467916µàÙ§C2- _wandbruntime‚5054  _runtime‚5054Ãó³£ç'‚áO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22125121 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43292959 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52924588 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9336021 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6246495817216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52924588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9336024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52914445 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚824765397328384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚824765755791872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12789100 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚819967338458085 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22125405 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚818554631900672 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184182693328896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6201733842031 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96217547 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12789381 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845300900442793e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚818554648940032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43292959 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43292957 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819967696058719 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813765962216688 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96207402 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚818554631900672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6201393280757 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184182817210368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6210765427712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177936216123904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚824765755791872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6211106851840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6211106851840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177936197511680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚824765397328384 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128Žc ƒæO F@DeviceStatsMonitor.on_€J5Ì'train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184182817210368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6210765427712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177936216123904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚824765755791872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6211106851840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6211106851840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177936197511680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚824765397328384 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22125121 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43292959 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52924588 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9336021 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6246495817216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52924588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9336024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52914445 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚824765397328384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚824765755791872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12789100 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚819967338458085 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22125405 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚818554631900672 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184182693328896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6201733842031 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96217547 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12789381 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845300900442793e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚818554648940032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43292959 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43292957 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819967696058719 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813765962216688 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96207402 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚818554631900672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6201393280757 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344¶|@ œ2„- _wandbruntime‚5054  _runtime‚5054oäÝ ð&…êM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52934258 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6212216957952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43300752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96224851 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6211875526656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚824914781929472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52934258 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96235010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43300752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚439 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚824915140400128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6212216957952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6247718586880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12791598 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚820116235114555 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22129076 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚818702923442176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9337755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚818702923442176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6211875526656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177969377781760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9337758 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚5054.934251043 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚813913375844800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52924101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43300750 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚820115877513809 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184217114980864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚824915140400128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12791321 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6202842230395 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6202501669009 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚813913392884160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96235010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚5349 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚818702906402816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52924101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.784530090665856e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22129356 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43300750 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚818702906402816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184216993203712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6247615421952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177969396393984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344Úqíõ&†ïM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6211875526656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚824914781929472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52934258 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96235010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43300752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚439 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚824915140400128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6212216957952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6247718586880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12791598 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚820116235114555 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22129076 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚818702923442176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9337755 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚818702923442176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6211875526656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177969377781760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9337758 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚5054.934251043 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚813913375844800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52924101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43300750 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚820115877513809 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184217114980864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚824915140400128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12791321 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6202842230395 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6202501669009 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚813913392884160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96235010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚5349 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚818702906402816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52924101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.784530090665856e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22129356 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43300750 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚818702906402816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184216993203712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6247615421952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177969396393984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52934258 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6212216957952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43300752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96224851 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180·7•Ö2‡- _wandbruntime‚5054  _runtime‚5054Eºˆ /loss/FRt2/detection_loss‚0.5387013209105749 1loss/SRqqt1/detection_loss‚0.5931196167562462 /loss/FBt1/detection_loss‚0.4162527781723301 $ _timestamp‚1.784530090667039e+09 _step‚440 1loss/FBt2/assignment_loss‚0.10153981045780425 trainer/global_step‚5349 0loss/FRt2/assignment_loss‚1.8043776822167557 0loss/SRqqt2/assignment_loss‚0.60784892815983 3loss/FBt1/assignment_loss‚0.0072117599695271095 /loss/FRt1/detection_loss‚1.4017639285667751 1loss/SRqqt2/detection_loss‚1.0514523771413777 /loss/FBt2/detection_loss‚0.6164324091514573 _runtime‚5054.934842574 0loss/FRt1/assignment_loss‚0.6575294370620938 2loss/SRqqt1/assignment_loss‚0.5276315778976408 %loss/total_loss‚8.323861626462413 epoch‚13S'îô ‰† 1loss/SRqqt1/detection_loss‚0.5931196167562462 /loss/FBt1/detection_loss‚0.4162527781723301 $ _timestamp‚1.784530090667039e+09 _step‚440 1loss/FBt2/assignment_loss‚0.10153981045780425 trainer/global_step‚5349 0loss/FRt2/assignment_loss‚1.8043776822167557 0loss/SRqqt2/assignment_loss‚0.60784892815983 3loss/FBt1/assignment_loss‚0.0072117599695271095 /loss/FRt1/detection_loss‚1.4017639285667751 1loss/SRqqt2/detection_loss‚1.0514523771413777 /loss/FBt2/detection_loss‚0.6164324091514573 _runtime‚5054.934842574 0loss/FRt1/assignment_loss‚0.6575294370620938 2loss/SRqqt1/assignment_loss‚0.5276315778976408 %loss/total_loss‚8.323861626462413 epoch‚13 /loss/FRt2/detection_loss‚0.5387013209105749¸Ú “ˆŠ‚(:ý ª‰÷Ò¯àÀÞ network.sent‚ 8.611369e+06 network.recv‚ 1.304842e+06! disk.dm-0.out‚42214.716796875 disk.dm-1.out‚7593.845703125! disk.nvme0c0n1.in‚ 10.04296875 disk.rbd1.out‚0 disk.nvme1n1.in‚ 3.08203125( proc.memory.availableMB‚ 490.85546875) disk./.usagePercent‚76.15989035299053 disk.md0.in‚0 disk.md0.out‚ 134.4765625 disk.dm-0.in‚13.125# disk.nvme0n1.out‚32629.83203125 disk.rbd0.out‚0" disk.nvme1c1n1.out‚ 17467.15625 disk.nvme1n1.out‚ 17467.15625 disk.dm-1.in‚0 disk.rbd1.in‚0$ memory_percent‚97.00405597686768" proc.memory.rssMB‚ 2900.4765625) proc.memory.percent‚17.70310401916504$ disk./.usageGB‚1169.244716644287% disk.nvme0c0n1.out‚32629.83203125 disk.nvme0n1.in‚ 10.04296875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 3.08203125 cpu‚25.68977962522162 proc.cpu.threads‚17z§´‹‚(:“ ª‰÷ 0©hÒÁ»‹ß gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.581' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.19366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_؆y˜Œ‚(: ¹‰÷ÒÖñšÞ network.recv‚ 1.308709e+06 disk.nvme1c1n1.in‚ 3.08203125 disk.dm-0.in‚13.125 disk.dm-0.out‚ 42220.515625 disk.nvme0n1.in‚ 10.04296875 network.sent‚ 8.662849e+06( disk.nvme1c1n1.out‚17468.40771484375 disk.md0.out‚ 134.67578125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚32635.0068359375$ memory_percent‚97.00345993041992" proc.memory.rssMB‚ 2900.4765625 cpu‚25.690402708196608 disk.rbd1.in‚0 disk.rbd1.out‚0& disk.nvme1n1.out‚17468.40771484375% disk.nvme0n1.out‚32635.0068359375 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 490.953125) proc.memory.percent‚17.70310401916504) disk./.usagePercent‚76.15988811672443% disk./.usageGB‚1169.2446823120117 disk.nvme1n1.in‚ 3.08203125 disk.md0.in‚0" disk.dm-1.out‚7594.03369140625! disk.nvme0c0n1.in‚ 10.04296875 proc.cpu.threads‚17¡S~Z“‚(:ˆ ¹‰÷ÒÂïâÞ gpu.0.gpu‚59.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.909' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.303 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÅ׎‚(:… ȉ÷Ò‰Ë•Þ disk.nvme0n1.in‚ 10.04296875 disk.nvme0n1.out‚ 32636.53125( disk.nvme1c1n1.out‚17469.00927734375 disk.rbd1.out‚0& disk.nvme1n1.out‚17469.00927734375$ memory_percent‚97.00398445129395 network.sent‚ 8.665771e+06" disk.dm-0.out‚42221.6533203125! disk.nvme0c0n1.in‚ 10.04296875 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.in‚0) disk./.usagePercent‚76.14441141595361 disk.dm-0.in‚13.125" disk.dm-1.out‚7594.08447265625 disk.nvme1n1.in‚ 3.08203125' proc.memory.availableMB‚ 490.8671875) proc.memory.percent‚17.70310401916504 cpu‚25.69052937995903% disk./.usageGB‚1169.0070762634277 disk.md0.out‚ 135.109375 disk.dm-1.in‚0" disk.nvme0c0n1.out‚ 32636.53125 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2900.4765625 proc.cpu.threads‚17 network.recv‚ 1.311908e+06 disk.md0.in‚0€î“‚(:ˆ ȉ÷Ò³ÈðÞ gpu.0.gpu‚67.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.519' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.173 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~2- _wandbruntime‚5098  _runtime‚5098 Z¤±É‘à ' lr-AdamW/pg1‚0.00015158769239769317 ' lr-AdamW/pg2‚0.00015158769239769317 trainer/global_step‚5399 % _timestamp‚1.7845301339572194e+09 _runtime‚5098.225526755 _step‚441,_¨¨Î’È ' lr-AdamW/pg2‚0.00015158769239769317 trainer/global_step‚5399 % _timestamp‚1.7845301339572194e+09 _runtime‚5098.225526755 _step‚441 ' lr-AdamW/pg1‚0.00015158769239769317¹ûnç_2“- _wandbruntime‚5098  _runtime‚5098]’D‰ç'”áO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53399097 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821137495590288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6266334529536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚832235050000896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12903406 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚825968357007872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22326417 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43682607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43682607 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5098.230894033 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6256876494165 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12903693 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚832234691537408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6266334529536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53409240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9422724 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9422721 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827394712645727 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821137478550928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97091849 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6302465337344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚832234691537408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6302355873792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6266675953664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚832235050000896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43682609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185897694965248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚827394355045093 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚5399 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22326127 _step‚442 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53399097 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97081704 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43682609 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚825968374047232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7845301339579513e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232–ßìì'•æO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821137495590288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6266334529536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚832235050000896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12903406 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚825968357007872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22326417 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43682607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43682607 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5098.230894033 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6256876494165 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12903693 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚832234691537408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6266334529536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53409240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9422724 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9422721 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827394712645727 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821137478550928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97091849 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6302465337344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚832234691537408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6302355873792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6266675953664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚832235050000896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43682609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185897694965248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚827394355045093 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚5399 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22326127 _step‚442 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53399097 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97081704 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43682609 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚825968374047232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7845301339579513e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53399097 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452ºg÷L€2–- _wandbruntime‚5098  _runtime‚5098—6ðÊ!—ëM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43690402 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53418910 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22330376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12905615 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6267444628480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43690400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185931990564864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6303475404800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43690400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53418910 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97099153 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53408753 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22330070 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9424455 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚832384434609152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53408753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚821284909218400 _step‚443 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43690402 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚826116648549376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179628409897984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185931866690560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5098.842433085 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚826116631510016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97109312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826116631510016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚832384434609152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚827542894100817 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845301345740979e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6303580666880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179628409897984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185931866690560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5098.842433085 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚826116631510016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97109312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826116631510016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚832384434609152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚827542894100817 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845301345740979e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6303580666880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22330070 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9424455 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚832384434609152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53408753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚821284909218400»‚[Á;2™- _wandbruntime‚5098  _runtime‚5098"â›—š€ /loss/FRt1/assignment_loss‚0.764463709952595 1loss/SRqqt1/assignment_loss‚0.549680069358941 1loss/SRqqt1/detection_loss‚0.6100655727456687 %loss/total_loss‚8.478733947630293 trainer/global_step‚5399 % _timestamp‚1.7845301345751927e+09 0loss/FRt2/assignment_loss‚1.8328272041736864 2loss/FBt1/assignment_loss‚0.042538038236388874 2loss/FBt2/assignment_loss‚0.001258889220485755 .loss/FBt2/detection_loss‚0.803798191271545 epoch‚13 2loss/SRqqt2/assignment_loss‚0.5150460539432273 /loss/FRt1/detection_loss‚1.4875292805094809 1loss/SRqqt2/detection_loss‚0.9612434201589819 _runtime‚5098.842983096 /loss/FRt2/detection_loss‚0.5341503547252989 .loss/FBt1/detection_loss‚0.376133163333993 _step‚4443ègL ›… 1loss/SRqqt1/detection_loss‚0.6100655727456687 %loss/total_loss‚8.478733947630293 trainer/global_step‚5399 % _timestamp‚1.7845301345751927e+09 0loss/FRt2/assignment_loss‚1.8328272041736864 2loss/FBt1/assignment_loss‚0.042538038236388874 2loss/FBt2/assignment_loss‚0.001258889220485755 .loss/FBt2/detection_loss‚0.803798191271545 epoch‚13 2loss/SRqqt2/assignment_loss‚0.5150460539432273 /loss/FRt1/detection_loss‚1.4875292805094809 1loss/SRqqt2/detection_loss‚0.9612434201589819 _runtime‚5098.842983096 /loss/FRt2/detection_loss‚0.5341503547252989 .loss/FBt1/detection_loss‚0.376133163333993 _step‚444 /loss/FRt1/assignment_loss‚0.764463709952595 1loss/SRqqt1/assignment_loss‚0.549680069358941¼rÈJMžœ‚(:“ ׉÷ÒÒݾÞ disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-0.in‚13.125! disk.nvme0c0n1.in‚ 10.04296875 disk.rbd0.out‚0$ memory_percent‚97.00808525085449 disk.nvme1c1n1.in‚ 3.08203125& disk.nvme1n1.out‚17472.15283203125( disk.nvme0c0n1.out‚32652.06494140625' proc.memory.availableMB‚ 490.1953125" proc.memory.rssMB‚ 2900.4765625 cpu‚25.691188968970028 proc.cpu.threads‚17 network.recv‚ 1.315616e+06 disk.nvme1n1.in‚ 3.08203125# disk.dm-0.out‚42239.40087890625 disk.nvme0n1.in‚ 10.04296875) proc.memory.percent‚17.70310401916504 network.sent‚ 8.684676e+06) disk./.usagePercent‚76.14445663822374# disk./.usageGB‚1169.00777053833 disk.md0.out‚ 135.49609375 disk.dm-1.in‚0" disk.dm-1.out‚7594.16845703125& disk.nvme0n1.out‚32652.06494140625 disk.rbd0.in‚0( disk.nvme1c1n1.out‚17472.15283203125·HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚834821371632101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚839703985746432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13017519 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚839704344209920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6321903631360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚839703985746432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187612400229888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181254243131904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚5449 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚828509011924528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6312700268847 $ _timestamp‚1.784530177581538e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97956006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6321903631360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53893892 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53893892 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚834821729232735 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22526940 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44072257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚833382082115072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13017810 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833382099154432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44072259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚834821371632101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚839703985746432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13017519 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚839704344209920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6321903631360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚839703985746432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187612400229888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚c|ø—C0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181254243131904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚5449 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚828509011924528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6312700268847 $ _timestamp‚1.784530177581538e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97956006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6321903631360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53893892 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53893892 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚834821729232735 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22526940 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44072257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚833382082115072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13017810 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833382099154432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44072259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚839853728818176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6323355161600 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181287423401984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚447 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13019841 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359395936256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44080052 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚839853370347520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53903562 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53893405 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9511155 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6313808657211 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97983614 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44080052 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚834970268288571 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6313468095825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13020111 " _timestamp‚1.7845301782008e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22530996 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187646819338240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97983614 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22531269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚833530373656576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44080050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6359294868480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53893405 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _runtime‚5142.469089863 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚828656442592000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181287404789760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22530996 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187646819338240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97983614 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22531269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚833530373656576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44080050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6359294868480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53893405 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _runtime‚5142.469089863 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚828656442592000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181287404789760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚839853728818176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6323355161600 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181287423401984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚447 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13019841 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359395936256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44080052 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚839853370347520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53903562 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53893405 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9511155¿êÖä02«- _wandbruntime‚5142  _runtime‚5142@óÝò ¬„ 0loss/FRt2/assignment_loss‚1.8041946928538577 /loss/FBt1/detection_loss‚0.4221986911143176 epoch‚13 2loss/SRqqt1/assignment_loss‚0.5027659118840273 2loss/FBt2/assignment_loss‚0.001662898191087056 /loss/FRt1/detection_loss‚1.4625959209071926 1loss/SRqqt2/detection_loss‚0.9041009731133657 trainer/global_step‚5449 _runtime‚5142.469627044 0loss/FRt1/assignment_loss‚0.6357958856205703 /loss/FBt2/detection_loss‚0.9321630191499073 $loss/total_loss‚8.42793134600973 2loss/SRqqt2/assignment_loss‚0.5928451686514045 3loss/FBt1/assignment_loss‚0.0076066001196499755 /loss/FRt2/detection_loss‚0.5360817520738299 1loss/SRqqt1/detection_loss‚0.6259198323305213 % _timestamp‚1.7845301782018485e+09 _step‚448±;Á­‰ 2loss/SRqqt2/assignment_loss‚0.5928451686514045 3loss/FBt1/assignment_loss‚0.0076066001196499755 /loss/FRt2/detection_loss‚0.5360817520738299 1loss/SRqqt1/detection_loss‚0.6259198323305213 % _timestamp‚1.7845301782018485e+09 _step‚448 0loss/FRt2/assignment_loss‚1.8041946928538577 /loss/FBt1/detection_loss‚0.4221986911143176 epoch‚13 2loss/SRqqt1/assignment_loss‚0.5027659118840273 2loss/FBt2/assignment_loss‚0.001662898191087056 /loss/FRt1/detection_loss‚1.4625959209071926 1loss/SRqqt2/detection_loss‚0.9041009731133657 trainer/global_step‚5449 _runtime‚5142.469627044 0loss/FRt1/assignment_loss‚0.6357958856205703 /loss/FBt2/detection_loss‚0.9321630191499073 $loss/total_loss‚8.42793134600973ÀWPïA—®‚(:Œ „Š÷Òó¹ªÞ disk.nvme1n1.in‚ 3.08203125% disk.nvme1n1.out‚17473.9814453125# memory_percent‚97.0071792602539% proc.memory.availableMB‚ 490.34375 network.sent‚ 8.757015e+06 network.recv‚ 1.326514e+06! disk.dm-1.out‚7594.5166015625( disk.nvme0c0n1.out‚32655.25244140625" proc.memory.rssMB‚ 2900.4765625 cpu‚25.69436739662671 proc.cpu.threads‚17) disk./.usagePercent‚76.14445589280172 disk.md0.in‚0 disk.nvme0n1.in‚ 10.04296875 disk.rbd1.out‚0) proc.memory.percent‚17.70310401916504% disk./.usageGB‚1169.0077590942383 disk.dm-0.in‚13.125# disk.dm-0.out‚42241.65478515625! disk.nvme0c0n1.in‚ 10.04296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125' disk.nvme1c1n1.out‚17473.9814453125 disk.md0.out‚ 136.609375 disk.dm-1.in‚0& disk.nvme0n1.out‚32655.25244140625 disk.rbd1.in‚0fFWꞯ‚(:“ „Š÷ÒûõñÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.985' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.32833333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇeŸ~™°‚(:Ž “Š÷Ò‰ŠùÝ disk.dm-0.in‚13.125 disk.dm-1.in‚0! disk.dm-1.out‚7594.8349609375( disk.nvme0c0n1.out‚32754.98681640625& disk.nvme0n1.out‚32754.98681640625 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.in‚0 network.recv‚ 1.330892e+06% disk./.usageGB‚1169.2925567626953 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚0' proc.memory.availableMB‚ 489.9765625" proc.memory.rssMB‚ 2900.4765625 cpu‚25.695401965273398 disk.md0.in‚0! disk.nvme0c0n1.in‚ 10.04296875& disk.nvme1c1n1.out‚17481.806640625$ memory_percent‚97.00942039489746 proc.cpu.threads‚17 network.sent‚ 8.80742e+06) disk./.usagePercent‚76.16300646557208# disk.dm-0.out‚42348.28076171875 disk.nvme0n1.in‚ 10.04296875 disk.nvme1n1.in‚ 3.08203125$ disk.nvme1n1.out‚17481.806640625) proc.memory.percent‚17.70310401916504 disk.md0.out‚ 136.8984375û4B뱂(:’ “Š÷Òóá¿Þ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.663' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.221000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D?©›²‚(: ¢Š÷Ò“õÝ' proc.memory.availableMB‚ 489.9765625" proc.memory.rssMB‚ 2900.4765625 cpu‚25.695319270151526 network.sent‚ 8.810342e+06 network.recv‚ 1.333948e+06 disk.md0.in‚0 disk.dm-0.in‚13.125# disk.dm-0.out‚42432.10107421875! disk.nvme0c0n1.in‚ 10.04296875 disk.rbd1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1169.2455406188965 disk.md0.out‚ 137.58984375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚32812.88134765625 disk.rbd1.in‚0$ memory_percent‚97.00942039489746) disk./.usagePercent‚76.15994402337707! disk.dm-1.out‚7594.9482421875& disk.nvme0n1.out‚32812.88134765625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125& disk.nvme1c1n1.out‚17509.291015625$ disk.nvme1n1.out‚17509.291015625) proc.memory.percent‚17.70310401916504 disk.nvme0n1.in‚ 10.04296875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 3.08203125¢Åà“³‚(:ˆ ¢Š÷Ò“êÍÞ gpu.0.gpu‚81.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.491' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.497 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÜ(å2´- _wandbruntime‚5184  _runtime‚5184àüƒÜǵÁ _step‚449 & lr-AdamW/pg1‚0.0001505506601892761 & lr-AdamW/pg2‚0.0001505506601892761 trainer/global_step‚5499 % _timestamp‚1.7845302207213485e+09 _runtime‚5184.990079854§ÚÞzÌ¶Æ & lr-AdamW/pg2‚0.0001505506601892761 trainer/global_step‚5499 % _timestamp‚1.7845302207213485e+09 _runtime‚5184.990079854 _step‚449 & lr-AdamW/pg1‚0.0001505506601892761Á'«±¯2·- _wandbruntime‚5184  _runtime‚5184B&ôQ¸áO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22728659 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22728364 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189327434659328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44461907 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allo´¦i–"cated‚1180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚840795807222272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13132535 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚835880545298128 trainer/global_step‚5499 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6377472733184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98840453 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182913238023680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842248388219109 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54378544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13132243 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414178023424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚840795824261632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9596121 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842248745819743 % _timestamp‚1.7845302207220545e+09 _runtime‚5184.990145774 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98830308 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚840795824261632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54368401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6377472733184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6368183482255 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44461909 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54368401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6377814157312 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847173279955456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Bd[ì'¹æO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚847173279955456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9596124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182913256635904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚450 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚847173638418944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54378544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6414076948480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚847173638418944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6367842920981 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847173279955456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22728659 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22728364 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189327434659328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44461907 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚840795807222272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13132535 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚835880545298128 trainer/global_step‚5499 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6377472733184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98840453 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182913238023680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842248388219109 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54378544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13132243 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414178023424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚840795824261632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9596121 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842248745819743 % _timestamp‚1.7845302207220545e+09 _runtime‚5184.990145774 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98830308 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚840795824261632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54368401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6377472733184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6368183482255 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44461909 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54368401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6377814157312 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6378924263424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98857916 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6378582832128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54388214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182946436905984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚847323023027200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚836027975965600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22732321 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189361610255360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚451 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189361732032512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54378057 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13134748 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6415295126528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6369291870619 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44469702 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13134466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚842397284875579 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44469700 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9597858 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98857916 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98847757 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6378582832128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚840944081724416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22732606 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6368951309233 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚5185.605105364 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54388214 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6378924263424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚836027993004960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚847323023027200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44469700 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182946418293760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344y„ [8¼ðM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚840944081724416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22732606 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6368951309233 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚5185.605105364 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54388214 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6378924263424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/reques2Ê2;¾ted_bytes.large_pool.allocated‚836027993004960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚847323023027200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44469700 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182946418293760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6378924263424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98857916 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6378582832128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54388214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182946436905984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚847323023027200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚836027975965600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22732321 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189361610255360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚451 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189361732032512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54378057 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13134748 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6415295126528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6369291870619 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44469702 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13134466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚842397284875579 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44469700 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9597858 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98857916 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98847757 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6378582832128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ÃŽ•÷Ù2½- _wandbruntime‚5185  _runtime‚5185ëÝÿÌ ¾ƒ 2loss/SRqqt1/assignment_loss‚0.5381136753697351 2loss/SRqqt2/assignment_loss‚0.5618028245477695 /loss/FRt2/detection_loss‚0.4986799916629803 0loss/SRqqt1/detection_loss‚0.633018449440076 epoch‚13 3loss/FBt2/assignment_loss‚0.0015075552812757248 /loss/FBt2/detection_loss‚1.0376421677431575 trainer/global_step‚5499 /loss/FRt1/detection_loss‚1.4625913089877296 /loss/FBt1/detection_loss‚0.5289513197499133 % _timestamp‚1.7845302213378048e+09 _runtime‚5185.605658355 0loss/FRt1/assignment_loss‚0.7045048991655973 /loss/FRt2/assignment_loss‚1.790319334514928 2loss/FBt1/assignment_loss‚0.009170025689083684 1loss/SRqqt2/detection_loss‚0.8722222903395384 %loss/total_loss‚8.638523842491786 _step‚452¥GÆ¡¿ˆ /loss/FRt1/detection_loss‚1.4625913089877296 /loss/FBt1/detection_loss‚0.5289513197499133 % _timestamp‚1.7845302213378048e+09 _runtime‚5185.605658355 0loss/FRt1/assignment_loss‚0.7045048991655973 /loss/FRt2/assignment_loss‚1.790319334514928 2loss/FBt1/assignment_loss‚0.009170025689083684 1loss/SRqqt2/detection_loss‚0.8722222903395384 %loss/total_loss‚8.638523842491786 _step‚452 2loss/SRqqt1/assignment_loss‚0.5381136753697351 2loss/SRqqt2/assignment_loss‚0.5618028245477695 /loss/FRt2/detection_loss‚0.4986799916629803 0loss/SRqqt1/detection_loss‚0.633018449440076 epoch‚13 3loss/FBt2/assignment_loss‚0.0015075552812757248 /loss/FBt2/detection_loss‚1.0376421677431575 trainer/global_step‚5499Ä3±?p—À‚(:Œ ±Š÷ÒÒ¾²Þ disk.nvme1n1.in‚ 3.08203125& disk.nvme1n1.out‚17541.59326171875$ memory_percent‚97.00989723205566" proc.memory.rssMB‚ 2900.4765625 cpu‚25.6946075005984 network.recv‚ 1.337457e+06) disk./.usagePercent‚76.15992389698212$ disk./.usageGB‚1169.245231628418 disk.dm-0.in‚13.125 disk.dm-1.in‚0& disk.nvme0c0n1.out‚32853.451171875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125 network.sent‚ 8.829534e+06 disk.md0.in‚0# disk.dm-0.out‚42503.50146484375! disk.dm-1.out‚7595.0810546875! disk.nvme0c0n1.in‚ 10.04296875 disk.nvme0n1.in‚ 10.04296875 disk.rbd1.in‚0 disk.rbd1.out‚0$ disk.nvme0n1.out‚32853.451171875 disk.rbd0.in‚0( disk.nvme1c1n1.out‚17541.59326171875' proc.memory.availableMB‚ 489.8984375) proc.memory.percent‚17.70310401916504 proc.cpu.threads‚17 disk.md0.out‚ 138.2265625ó–>{ŸÁ‚(:” ±Š÷ÒÂå÷Þ gpu.0.gpu‚86.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚123.205' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.068333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µkUšÂ‚(: ÀŠ÷ÒŠ·žÞ# disk.dm-0.out‚42504.07958984375! disk.dm-1.out‚7595.1787109375 network.recv‚ 1.341527e+06$ disk.nvme1n1.out‚17542.130859375 disk.dm-0.in‚13.125 disk.nvme0n1.in‚ 10.04296875& disk.nvme0n1.out‚32854.32861328125( proc.memory.availableMB‚ 489.89453125) proc.memory.percent‚17.70310401916504 cpu‚25.69405008471398 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125" proc.memory.rssMB‚ 2900.4765625 proc.cpu.threads‚17 network.sent‚ 8.879838e+06% disk./.usageGB‚1169.2453231811523 disk.rbd1.in‚0 disk.md0.out‚ 138.56640625 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.04296875( disk.nvme0c0n1.out‚32854.32861328125$ memory_percent‚97.00992107391357( disk./.usagePercent‚76.1599298603584 disk.rbd0.in‚0& disk.nvme1c1n1.out‚17542.130859375 disk.rbd1.out‚0 disk.nvme1n1.in‚ 3.08203125 disk.md0.in‚0@'¡ZžÃ‚(:“ ÀŠ÷Ò“¥æÞ gpu.0.gpu‚93.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.333' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.44433333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÔz:ŸÄ‚(:” ÏŠ÷Ò°ÆÞ network.sent‚ 8.884713e+06 network.recv‚ 1.344815e+06& disk.nvme0n1.out‚32855.51513671875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.in‚0& disk.nvme1n1.out‚17542.37744140625 disk.md0.in‚0) disk./.usagePercent‚76.15994601116917 disk.nvme1n1.in‚ 3.08203125 disk.md0.out‚ 138.640625# disk.dm-0.out‚42505.26123046875 disk.dm-1.in‚0" disk.dm-1.out‚7595.26416015625! disk.nvme0c0n1.in‚ 10.04296875( disk.nvme0c0n1.out‚32855.51513671875 disk.nvme0n1.in‚ 10.04296875 disk.rbd0.in‚0( disk.nvme1c1n1.out‚17542.37744140625 disk.dm-0.in‚13.125$ memory_percent‚97.01073169708252" proc.memory.rssMB‚ 2900.4765625) proc.memory.percent‚17.70310401916504 cpu‚25.693352175409863% disk./.usageGB‚1169.2455711364746 disk.rbd1.out‚0( proc.memory.availableMB‚ 489.76171875 proc.cpu.threads‚17.,3’Å‚(:‡ ÏŠ÷Ò Ú×Þ gpu.0.gpu‚32.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.144' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S‡2Æ- _wandbruntime‚5228  _runtime‚5228|Q-ÉÇà ' lr-AdamW/pg2‚0.00015002623503659838 trainer/global_step‚5549 % _timestamp‚1.7845302640231905e+09 _runtime‚5228.291332757 _step‚453 ' lr-AdamW/pg1‚0.00015002623503659838;&ÌÎÈÈ ' lr-AdamW/pg2‚0.00015002623503659838 trainer/global_step‚5549 % _timestamp‚1.7845302640231905e+09 _runtime‚5228.291332757 _step‚453 ' lr-AdamW/pg1‚0.00015002623503659838Åx³Þ<2É- _wandbruntime‚5228  _runtime‚5228L¦k«æ'ÊàO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44851559 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848209549368832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚5228.295601425 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚854642932627968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚848209549368832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54853053 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9682821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚849675404806117 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843252095711088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184572270139904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99714755 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22930413 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6433383259136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191042275999232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6470005859328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54863196 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7845302640241997e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 trainer/global_step‚5549 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚854642932627968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚854642574164480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54853053 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚854642574164480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44851557 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚454 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44851557 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849675762406751 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13247306 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184572251527680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6433041835008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚848209532329472 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191042158409216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44851559 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13247589 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9682824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54863196 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6433041835008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22930127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6469906881536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843252078671728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¼Ò/“ 'ËåO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191042158409216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44851559 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13247589 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9682824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54863196 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6433041835008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22930127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6469906881536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843252078671728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44851559 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848209549368832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚5228.295601425 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚854642932627968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚848209549368832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54853053 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9682821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚849675404806117 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843252095711088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184572270139904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99714755 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22930413 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6433383259136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191042275999232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6470005859328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54863196 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7845302640241997e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 trainer/global_step‚5549 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚854642932627968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚854642574164480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54853053 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚854642574164480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44851557 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚454 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44851557 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849675762406751 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13247306 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184572251527680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6433041835008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚848209532329472 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99732218 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6424775084027 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚849823943861841 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843399509339200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚455 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚854792317236224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848357823870976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99722059 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6471125216256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854792317236224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54872866 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54872866 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99732218 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54862709 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6434151933952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54862709 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184605450409984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184605431797760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5228.907922017 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22934135 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13249861 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845302646396065e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191076575626240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6471024148480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚848357806831616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99722059 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303Seú£ö&ÎðM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6471024148480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚848357806831616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99722059 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22934419 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6434151933952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9684555 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99732218 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6424775084027 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚849823943861841 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843399509339200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚455 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚854792317236224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848357823870976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99722059 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6471125216256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854792317236224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54872866 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54872866 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99732218 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54862709 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6434151933952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54862709 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184605450409984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184605431797760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5228.907922017 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22934135 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13249861 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845302646396065e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191076575626240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ç]ì©2Ï- _wandbruntime‚5228  _runtime‚5228ŠºÄ· Є /loss/FRt1/detection_loss‚1.5327904847454767 /loss/FRt2/detection_loss‚0.5307074926135197 1loss/SRqqt1/detection_loss‚0.6259530139522221 /loss/FBt2/detection_loss‚0.9580895240035128 $loss/total_loss‚8.82127679479752 % _timestamp‚1.7845302646406515e+09 0loss/FRt1/assignment_loss‚0.7540928550165703 2loss/SRqqt1/assignment_loss‚0.5110870282081379 _step‚456 1loss/SRqqt2/assignment_loss‚0.560871357600501 epoch‚13 _runtime‚5228.908425678 0loss/FRt2/assignment_loss‚1.8936220043501863 3loss/FBt1/assignment_loss‚0.0022166728420407846 3loss/FBt2/assignment_loss‚0.0013622506904689528 1loss/SRqqt2/detection_loss‚1.0315284283018924 /loss/FBt1/detection_loss‚0.4189556824729901 trainer/global_step‚5549Bó{щ /loss/FRt1/detection_loss‚1.5327904847454767 /loss/FRt2/detection_loss‚0.5307074926135197 1loss/SRqqt1/detection_loss‚0.6259530139522221 /loss/FBt2/detection_loss‚0.9580895240035128 $loss/total_loss‚8.82127679479752 % _timestamp‚1.7845302646406515e+09 0loss/FRt1/assignment_loss‚0.7540928550165703 2loss/SRqqt1/assignment_loss‚0.5110870282081379 _step‚456 1loss/SRqqt2/assignment_loss‚0.560871357600501 epoch‚13 _runtime‚5228.908425678 0loss/FRt2/assignment_loss‚1.8936220043501863 3loss/FBt1/assignment_loss‚0.0022166728420407846 3loss/FBt2/assignment_loss‚0.0013622506904689528 1loss/SRqqt2/detection_loss‚1.0315284283018924 /loss/FBt1/detection_loss‚0.4189556824729901 trainer/global_step‚5549Èá巜҂(:‘ ÞŠ÷Ò¹Þ‚Þ disk.rbd1.in‚0 disk.nvme1n1.in‚ 3.08203125" disk.dm-1.out‚7595.34228515625! disk.nvme0c0n1.in‚ 10.04296875( proc.memory.availableMB‚ 489.65234375) proc.memory.percent‚17.70310401916504 network.recv‚ 1.348096e+06& disk.nvme1n1.out‚17542.81494140625 disk.md0.out‚ 138.87109375 disk.dm-0.in‚13.125" disk.dm-0.out‚42506.0947265625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 10.04296875" memory_percent‚97.011399269104 network.sent‚ 8.887677e+06( disk.nvme1c1n1.out‚17542.81494140625 disk.md0.in‚0" proc.memory.rssMB‚ 2900.4765625 cpu‚25.692753271398317 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0' disk.nvme0c0n1.out‚32856.5205078125% disk.nvme0n1.out‚32856.5205078125 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.15994998675335% disk./.usageGB‚1169.2456321716309¿E(úžÓ‚(:“ ÞŠ÷ÒëÿÅÞ gpu.0.gpu‚78.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚291.448' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.14933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]C•˜Ô‚(: íŠ÷Ò®ú‹Þ disk.nvme1c1n1.in‚ 3.08203125 disk.nvme1n1.in‚ 3.08203125 disk.md0.in‚0 disk.dm-1.in‚0& disk.nvme1n1.out‚17543.89697265625" memory_percent‚97.011399269104) proc.memory.percent‚17.70310401916504 cpu‚25.692108576726532! disk.nvme0c0n1.in‚ 10.04296875 disk.rbd1.in‚0 disk.md0.out‚ 139.453125 disk.dm-0.out‚42508.29296875( proc.memory.availableMB‚ 489.65234375" proc.memory.rssMB‚ 2900.4765625 proc.cpu.threads‚17) disk./.usagePercent‚76.15995023522737% disk./.usageGB‚1169.2456359863281' disk.nvme0c0n1.out‚32859.0556640625 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚17543.89697265625 disk.rbd1.out‚0 disk.dm-0.in‚13.125 network.sent‚ 8.950081e+06 network.recv‚ 1.352571e+06" disk.dm-1.out‚7595.52490234375 disk.nvme0n1.in‚ 10.04296875% disk.nvme0n1.out‚32859.0556640625ÓÿHŒœÕ‚(:‘ íŠ÷Ò¬€ÛÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.073' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.69099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bJ²–Ö‚(:‹ üŠ÷Ò»ù—Þ% disk.nvme0n1.out‚32859.9326171875 disk.rbd1.in‚0& disk.nvme1n1.out‚17544.62158203125% disk./.usageGB‚1169.2462005615234 disk.nvme0n1.in‚ 10.04296875( proc.memory.availableMB‚ 489.65234375 proc.cpu.threads‚17 disk.dm-0.in‚13.125" disk.dm-1.out‚7595.60302734375! disk.nvme0c0n1.in‚ 10.04296875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0 disk.nvme1n1.in‚ 3.08203125) proc.memory.percent‚17.70310401916504 network.recv‚ 1.355986e+06 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚17544.62158203125" memory_percent‚97.011399269104" proc.memory.rssMB‚ 2900.4765625 cpu‚25.691420027294928 network.sent‚ 8.955058e+06( disk./.usagePercent‚76.1599870093811 disk.md0.out‚ 139.7890625 disk.dm-0.out‚ 42509.078125' disk.nvme0c0n1.out‚32859.9326171875î>9\ž×‚(:“ üŠ÷Òá®÷Þ gpu.0.gpu‚98.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚265.533' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.51100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&f Z2Ø- _wandbruntime‚5271  _runtime‚5271ÂȶÅÙ¿ trainer/global_step‚5599 % _timestamp‚1.7845303072923596e+09 _runtime‚ 5271.5604691 _step‚457 & lr-AdamW/pg1‚0.0001494979153446372 & lr-AdamW/pg2‚0.0001494979153446372Eì|åÊÚÄ & lr-AdamW/pg1‚0.0001494979153446372 & lr-AdamW/pg2‚0.0001494979153446372 trainer/global_step‚5599 % _timestamp‚1.7845303072923596e+09 _runtime‚ 5271.5604691 _step‚457ÉR”2Û- _wandbruntime‚5271  _runtime‚5271ñ€§ÜåO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚855623257436672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _step‚458 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6488610936832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45241207 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186231283643904 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845303072932346e+09 A7DeviceStatsMonitor.on_train_batch_start/act„"@žDive.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6479149909071 trainer/global_step‚5599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862111868373504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100589057 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55337705 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13361429 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9769524 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6488610936832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45241209 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚850623629084688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚855623274476032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45241207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862112226836992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55347848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9769521 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186231283643904 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845303072932346e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6479149909071 trainer/global_step‚5599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862111868373504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100589057 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55337705 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13361429 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9769524 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6488610936832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45241209 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚850623629084688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚855623274476032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45241207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862112226836992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55347848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9769521 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855771531938816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9771258 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192791310681088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13364060 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100606520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _step‚459 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100596361 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55357518 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6479917736049 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚850771059752160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚5599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45249000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9771255 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚850771042712800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45249002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186264445301760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚862261611445248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23135027 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6489721035776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23135318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚862261252974592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855771548978176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13363772 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45249000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6489721035776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6490062467072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5272.182617628 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚5599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45249000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9771255 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚850771042712800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45249002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186264445301760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚862261611445248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23135027 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6489721035776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23135318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚862261252974592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855771548978176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13363772 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45249000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6489721035776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6490062467072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5272.182617628 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855771531938816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9771258 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192791310681088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13364060 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100606520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _step‚459 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100596361 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55357518 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6479917736049 WGDevicejExStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚850771059752160 F&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5603841543197632 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861103296279907 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7018206715583801 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03447771072387695 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583134651184 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3192444145679474 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07225755602121353 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22230733931064606 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19703033566474915 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022320440039038658 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02981376089155674 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27267974615097046 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8266318440437317 *jet/accuracy_1_of_4‚0.9714263081550598 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6836089491844177 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046830303966999054 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10365026444196701 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.252818763256073 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6467362642288208 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_3_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7446008920669556 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 jet/accuracy_3_of_5‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20542985200881958 *jet/accuracy_1_of_3‚0.9034540057182312 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.673769474029541 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,jet/accuracy_4_of_4‚0.013899499550461769 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.649198055267334 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6987507343292236 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27247777581214905 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1748158186674118 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9957119226455688 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7542641758918762 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2614963948726654 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018816787749528885 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.68211829662323 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9948786497116089 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8836735486984253 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.692138135433197 +jet/accuracy_1_of_1‚0.34113454818725586 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11669271439313889 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021923765540122986 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984974205493927 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964388608932495 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.639572262763977 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.640853762626648 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05821019038558006 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3391050398349762 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6663088798522949 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8582010865211487 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6404619812965393 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5536880493164062 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6244627833366394 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6241936683654785 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2614963948726654 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 +jet/accuracy_3_of_4‚0.08763584494590759 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20811083912849426 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7643991112709045 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6962603330612183 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6300212144851685 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7661758661270142 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.220480278134346 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6505429744720459 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9372692108154297 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2728336453437805 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8391722440719604 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5660479664802551 _step‚461 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951264262199402 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.875447154045105 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898247718811035 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8896808624267578 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8993470072746277 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7719962000846863 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46721041202545166 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6431246995925903 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6404619812965393 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37748217582702637 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6907559633255005 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7507187724113464 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09915275871753693 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02242039330303669 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4776418209075928 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.918942928314209 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 (particle/accuracy‚0.8507409691810608 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8761149048805237 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6748892068862915 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6684005260467529 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6980729103088379 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4917208254337311 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8704649806022644 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018816787749528885³{°„ 1è˜{ @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5536880493164062 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6244627833366394 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6241936683654785 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2614963948726654 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 +jet/accuracy_3_of_4‚0.08763584494590759 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20811083912849426 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7643991112709045 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6962603330612183 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6300212144851685 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7661758661270142 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.220480278134346 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6505429744720459 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9372692108154297 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2728336453437805 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8391722440719604 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5660479664802551 _step‚461 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951264262199402 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.875447154045105 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898247718811035 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8896808624267578 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8993470072746277 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7719962000846863 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46721041202545166 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6431246995925903 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6404619812965393 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37748217582702637 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6907559633255005 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7507187724113464 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09915275871753693 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02242039330303669 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4776418209075928 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.918942928314209 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 (particle/accuracy‚0.8507409691810608 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8761149048805237 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6748892068862915 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6684005260467529 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6980729103088379 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4917208254337311 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8704649806022644 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018816787749528885 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5408333539962769 +particle/specificity‚0.9094011187553406 % _timestamp‚1.7845303240811799e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38330987095832825 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.207367941737175 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7584128379821777 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19438175857067108 jet/accuracy_2_of_5‚1 (loss/val_total_loss‚8.22626544379398 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9959888458251953 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25272202491760254 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6667953729629517 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2179863303899765 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7167572379112244 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0372106209397316 +jet/accuracy_2_of_2‚0.16022472083568573 +particle/sensitivity‚0.6362306475639343 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5025639533996582 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18896833062171936 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 epoch‚13 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7383857369422913 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7131937742233276 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7380774617195129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9881634712219238 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7180231809616089 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08673867583274841 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32706505060195923 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7584412693977356 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2608335316181183 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9763268232345581 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26980507373809814 +jet/accuracy_3_of_3‚0.07821759581565857 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3024274706840515 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21042215824127197 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021535957232117653 /particle/accuracy_4_of_4‚0.9423625469207764 trainer/global_step‚5599 _runtime‚5288.349745735 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7012291550636292 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.462130606174469 5validation_average_jet_accuracy‚0.377044141292572 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6254410743713379 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6562265753746033 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8266318440437317 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21397867798805237 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.766647458076477 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5128341913223267 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5603841543197632 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861103296279907 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7018206715583801 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03447771072387695 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583134651184 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3192444145679474 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07225755602121353 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22230733931064606 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19703033566474915 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022320440039038658 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02981376089155674 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27267974615097046 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8266318440437317 *jet/accuracy_1_of_4‚0.9714263081550598 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6836089491844177 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046830303966999054 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10365026444196701 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.252818763256073 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6467362642288208 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_3_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7446008920669556 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 jet/accuracy_3_of_5‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20542985200881958 *jet/accuracy_1_of_3‚0.9034540057182312 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.673769474029541 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,jet/accuracy_4_of_4‚0.013899499550461769 ?&Purity/*FRtöWæ“ *SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.649198055267334 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6987507343292236 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27247777581214905 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1748158186674118 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9957119226455688 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7542641758918762 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2614963948726654 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018816787749528885 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.68211829662323 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9948786497116089 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8836735486984253 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.692138135433197 +jet/accuracy_1_of_1‚0.34113454818725586 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11669271439313889 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021923765540122986 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984974205493927 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964388608932495 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.639572262763977 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.640853762626648 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05821019038558006 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3391050398349762 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6663088798522949 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8582010865211487 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6404619812965393 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752ÍÀ€»“é‚(:ˆ š‹÷ÒΞ©Þ disk.nvme1n1.in‚ 3.08203125$ disk.nvme1n1.out‚17545.716796875 disk.md0.in‚0 disk.md0.out‚ 140.14453125& proc.memory.availableMB‚ 464.421875& disk.nvme1c1n1.out‚17545.716796875 network.sent‚ 9.034318e+06 network.recv‚ 1.362832e+06% disk./.usageGB‚1169.2460021972656 disk.rbd0.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚42511.64306640625# memory_percent‚97.1653938293457( disk./.usagePercent‚76.1599740887325 disk.dm-1.out‚ 7595.97265625 disk.nvme0n1.in‚ 10.04296875& disk.nvme0n1.out‚32862.57470703125 proc.memory.rssMB‚ 2925.5625 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.04296875( disk.nvme0c0n1.out‚32862.57470703125 disk.rbd0.in‚0 disk.dm-0.in‚13.125* proc.memory.percent‚17.856216430664062 cpu‚25.69824511070961 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0’£žê‚(:’ š‹÷Ò’©ðÞ gpu.0.gpu‚19.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚287.201' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.73366666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã"D˜ë‚(: ©‹÷ÒÌçŠÞ$ memory_percent‚97.16541767120361 disk.dm-0.in‚13.125 disk.dm-1.out‚ 7596.03515625 disk.rbd0.out‚0 disk.nvme1n1.in‚ 3.08203125 disk.md0.in‚0* proc.memory.percent‚17.856216430664062 proc.cpu.threads‚17" disk.dm-0.out‚42513.6611328125 disk.nvme0n1.in‚ 10.04296875& disk.nvme0n1.out‚32864.70556640625 disk.rbd0.in‚0 disk.md0.out‚ 140.453125( proc.memory.availableMB‚ 464.41796875 cpu‚25.697548319289513 network.recv‚ 1.367207e+06% disk./.usageGB‚1169.2464561462402 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.04296875( disk.nvme1c1n1.out‚17546.34521484375 proc.memory.rssMB‚ 2925.5625 network.sent‚ 9.080334e+06( disk./.usagePercent‚76.1600036571399( disk.nvme0c0n1.out‚32864.70556640625 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.in‚0 disk.rbd1.out‚0& disk.nvme1n1.out‚17546.34521484375>:uržì‚(:“ ©‹÷Ò•·ÕÞ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.007' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.33566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aRÎ!í‚(:‚ ¸‹÷Òÿ™±Þ disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 3.08203125' disk.nvme1c1n1.out‚17546.7783203125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-1.out‚ 7596.0859375 proc.memory.rssMB‚ 2925.5625$ disk.nvme0n1.out‚32865.646484375 disk.nvme1n1.in‚ 3.08203125 disk.md0.out‚ 140.71484375 disk.dm-0.in‚13.125! disk.nvme0c0n1.in‚ 10.04296875* proc.memory.percent‚17.856216430664062( disk./.usagePercent‚76.1599783127907% disk./.usageGB‚1169.2460670471191 disk.rbd0.out‚0" disk.dm-0.out‚42514.4072265625 disk.dm-1.in‚0 cpu‚25.69694370784034 proc.cpu.threads‚17 network.sent‚ 9.085248e+06 network.recv‚ 1.37048e+06 disk.nvme0n1.in‚ 10.04296875% disk.nvme1n1.out‚17546.7783203125& disk.nvme0c0n1.out‚32865.646484375$ memory_percent‚97.16072082519531$ proc.memory.availableMB‚465.1875JÄ#Ãœî‚(:‘ ¸‹÷Ò´¤ß gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚219.839' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.27966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÉ@2ï- _wandbruntime‚5330  _runtime‚5330Ú`û)Éðà % _timestamp‚1.7845303666686761e+09 _runtime‚5330.937524607 _step‚462 ' lr-AdamW/pg1‚0.00014896573504637908 ' lr-AdamW/pg2‚0.00014896573504637908 trainer/global_step‚5649… ÎñÈ _runtime‚5330.937524607 _step‚462 ' lr-AdamW/pg1‚0.00014896573504637908 ' lr-AdamW/pg2‚0.00014896573504637908 trainer/global_step‚5649 % _timestamp‚1.7845303666686761e+09Î|):¦2ò- _wandbruntime‚5330  _runtime‚5330>*’ë'óåO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55913994 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6564497874432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871543824293888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187912857198080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚864979343458816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6564839298560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194515367563264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194515243681792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6554936971376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45753667 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45753665 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6602491752960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6602386483712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101657516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚864979326419456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚864979343458816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚871544182757376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55913994 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚864979326419456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9860337 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55903851 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚871544182757376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13517503 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859916401427152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101667661 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚866471355437888 trainer/global_step‚5649 % _timestamp‚1.7845303666695035e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344”ꪠð'ôêO trainer/global_step‚5649 % _timestamp‚1.7845303666695035e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55913994 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6564497874432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871543824293888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187912857198080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚864979343458816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6564839298560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194515367563264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194515243681792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6554936971376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45753667 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45753665 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6602491752960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6602386483712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101657516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚864979326419456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚864979343458816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚871544182757376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55913994 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚864979326419456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9860337 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55903851 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚871544182757376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13517503 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859916401427152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101667661 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚866471355437888ÏWã0®2õ- _wandbruntime‚5331  _runtime‚5331À¯ÏHmöîM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚866619894493724 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚866619536892978 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚5649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚865127617960960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187946056080384 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23381825 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6565607973376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55913507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23382110 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45761458 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚865127600921600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101685124 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101674965 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45761460 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45761458 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13520036 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6603504019968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860063832094624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚865127617960960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784530367286977e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55913507 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871693567365632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6565949404672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9862071 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6603607184896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 _step‚464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101674965 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚871693208894976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚865127600921600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ðÔ~¿ù&÷óM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45761460 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55923664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194549541488128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101685124 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55923664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6555704798354 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9862074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101674965 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45761460 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45761458 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13520036 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6603504019968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860063832094624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚865127617960960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784530367286977e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55913507 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871693567365632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6565949404672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9862071 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6603607184896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 _step‚464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101674965 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚871693208894976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚865127600921600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚866619894493724 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚866619536892978 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚5649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚865127617960960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187946056080384 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23381825 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6565607973376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55913507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23382110 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45761458 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚865127600921600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101685124 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚873898014424262 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56398646 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _runtime‚5373.815855454 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23579649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46143317 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46143315 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879013118502912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6620408400384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879013118502912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872393068566016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23579912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102541963 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56388503˺(@Z†éO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56398646 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879013476966400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13632872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867287934800752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_trY1·•$ain_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13632612 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196230297095680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6658407781376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879013118502912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6620408400384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879013118502912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872393068566016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23579912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102541963 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56388503 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189571870702080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚872393051526656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6620066976256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9947040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102531818 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚872393068566016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚873898014424262 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56398646 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _runtime‚5373.815855454 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23579649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46143317 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46143315 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102559426 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56408316 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9948774 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879162861574656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13635148 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102549267 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102549267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚874046911080732 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879162503104000 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46151110 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189605069584384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚5374.431658975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23583641 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6621177075200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879162861574656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56398159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23583922 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56408316 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46151108 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6621518506496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189605050972160 trainer/global_step‚5699 % _timestamp‚1.7845304101632965e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46151110 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚874046553479986 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867435365468224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6611528573148 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102559426 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659522924544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6611188011762 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272óDû«ù&‰óM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚5374.431658975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23583641 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6621177075200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879162861574656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56398159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23583922 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56408316 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46151108 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6621518506496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189605050972160 trainer/global_step‚5699 % _timestamp‚1.7845304101632965e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46151110 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚874046553479986 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867435365468224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6611528573148 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102559426 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659522924544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6611188011762 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9948771 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102559426 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56408316 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9948774 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879162861574656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13635148 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102549267 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102549267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚874046911080732 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879162503104000 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46151110 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189605069584384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240Ô°_¤x2Š- _wandbruntime‚5374  _runtime‚5374G²Û‹ /loss/FRt2/detection_loss‚0.5452427258311159 %loss/total_loss‚8.336597925496658 _step‚469 .loss/FRt1/assignment_loss‚0.64696086987917 1loss/SRqqt2/assignment_loss‚0.556361856739386 0loss/FBt1/detection_loss‚0.49783816518650664 epoch‚14 1loss/FBt2/assignment_loss‚0.12713547686871607 1loss/SRqqt1/detection_loss‚0.5235872195004343 /loss/FBt2/detection_loss‚0.8055590794857351 2loss/SRqqt1/assignment_loss‚0.5341458568230496 1loss/SRqqt2/detection_loss‚0.9437489305065329 trainer/global_step‚5699 % _timestamp‚1.7845304101647403e+09 _runtime‚5374.432539356 0loss/FRt2/assignment_loss‚1.8026993550525106 2loss/FBt1/assignment_loss‚0.016328195572067074 /loss/FRt1/detection_loss‚1.3369901940514346U¡Š Œ† trainer/global_step‚5699 % _timestamp‚1.7845304101647403e+09 _runtime‚5374.432539356 0loss/FRt2/assignment_loss‚1.8026993550525106 2loss/FBt1/assignment_loss‚0.016328195572067074 /loss/FRt1/detection_loss‚1.3369901940514346 /loss/FRt2/detection_loss‚0.5452427258311159 %loss/total_loss‚8.336597925496658 _step‚469 .loss/FRt1/assignment_loss‚0.64696086987917 1loss/SRqqt2/assignment_loss‚0.556361856739386 0loss/FBt1/detection_loss‚0.49783816518650664 epoch‚14 1loss/FBt2/assignment_loss‚0.12713547686871607 1loss/SRqqt1/detection_loss‚0.5235872195004343 /loss/FBt2/detection_loss‚0.8055590794857351 2loss/SRqqt1/assignment_loss‚0.5341458568230496 1loss/SRqqt2/detection_loss‚0.9437489305065329Õ©÷2ž‚(:“ ô‹÷Ò¾øÊÞ! disk.nvme0c0n1.in‚ 10.04296875( disk.nvme0c0n1.out‚32872.22509765625 disk.rbd1.in‚0 disk.rbd1.out‚0( proc.memory.availableMB‚ 465.12109375 disk.md0.in‚0# disk.dm-0.out‚42520.10693359375 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme1n1.out‚17549.51513671875* proc.memory.percent‚17.856216430664062 proc.cpu.threads‚17 network.recv‚ 1.384738e+06( disk./.usagePercent‚76.1599552047076% disk./.usageGB‚1169.2457122802734 disk.md0.out‚ 142.19921875 disk.nvme0n1.in‚ 10.04296875& disk.nvme0n1.out‚32872.22509765625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125" disk.dm-1.out‚7596.50634765625( disk.nvme1c1n1.out‚17549.51513671875 disk.nvme1n1.in‚ 3.08203125$ memory_percent‚97.16112613677979 proc.memory.rssMB‚ 2925.5625 cpu‚25.694383723669493 network.sent‚ 9.173498e+06 disk.dm-0.in‚13.125KGV>žŽ‚(:“ ô‹÷Òèí’ß gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.047' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.01566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½M¼…‚(:„ ƒŒ÷Ò²ÖæÞ cpu‚25.693750943430228% disk./.usageGB‚1169.2457885742188 disk.rbd1.in‚0" disk.nvme1n1.out‚ 17553.4921875 disk.md0.out‚ 145.98046875# memory_percent‚97.1611499786377* proc.memory.percent‚17.856216430664062 proc.cpu.threads‚17 network.sent‚ 9.225001e+06 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0 disk.dm-0.in‚3êLè 13.125 disk.nvme0n1.in‚ 10.04296875 network.recv‚ 1.389325e+06 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 17553.4921875 disk.nvme1n1.in‚ 3.08203125 disk.md0.in‚0" disk.dm-0.out‚42521.0458984375 disk.dm-1.in‚0! disk.dm-1.out‚7596.5830078125) disk./.usagePercent‚76.15996017418783 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 10.04296875& disk.nvme0c0n1.out‚32876.892578125$ disk.nvme0n1.out‚32876.892578125' proc.memory.availableMB‚ 465.1171875 proc.memory.rssMB‚ 2925.5625·—Hž‚(:“ ƒŒ÷Òî×–Þ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚253.907' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.63566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`§z—‘‚(:Œ ’Œ÷ÒÞ¡çÝ disk.dm-0.in‚ 13.1484375! disk.nvme0c0n1.in‚ 10.06640625& disk.nvme0c0n1.out‚32908.974609375 disk.rbd0.out‚0$ memory_percent‚97.16136455535889( proc.memory.availableMB‚ 465.08203125 network.sent‚ 9.227965e+06( disk./.usagePercent‚76.1797039191627' disk.nvme1c1n1.out‚17554.9443359375 disk.rbd1.in‚0 disk.nvme1n1.in‚ 3.08203125% disk.nvme1n1.out‚17554.9443359375* proc.memory.percent‚17.856216430664062% disk./.usageGB‚1169.5489044189453 disk.md0.in‚0 disk.dm-0.out‚42553.30859375! disk.dm-1.out‚7596.7392578125 disk.nvme0n1.in‚ 10.06640625$ disk.nvme0n1.out‚32908.974609375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 3.08203125 network.recv‚ 1.392636e+06 disk.dm-1.in‚0 disk.rbd1.out‚0 proc.memory.rssMB‚ 2925.5625 cpu‚25.693087624856126 proc.cpu.threads‚17 disk.md0.out‚ 146.5078125Œ—ã“’‚(:ˆ ’Œ÷Òœó¿Þ gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.574' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å\•*2“- _wandbruntime‚5416  _runtime‚5416.,×ðɔà _step‚470 ' lr-AdamW/pg1‚0.00014788992960052073 ' lr-AdamW/pg2‚0.00014788992960052073 trainer/global_step‚5749 % _timestamp‚1.7845304526639376e+09 _runtime‚5416.932002424õ2–- _wandbruntime‚5416  _runtime‚5416A.Dí'—çO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚879806776633856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 % _timestamp‚1.7845304526645806e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚886482412711936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚5749 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚879806793673216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13747308 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191230884206080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚881325031011270 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚879806776633856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675636078080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197945049493504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6675977502208 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚881325388611904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874659468174352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6665562836918 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5416.933470997 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6714268459520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103406120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56873155 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197945171277824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56883298 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665903398192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6675636078080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6675977502208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23781048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚879806793673216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191230902818304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46532967 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6714165287424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874659485213712 _step‚471 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚886482771175424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103416265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚886482412711936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13747033 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56883298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10033737 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46532965 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚886482771175424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚886482412711936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚5749 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚879806793673216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13747308 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191230884206080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚881325031011270 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚879806776633856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675636078080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197945049493504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6675977502208 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚881325388611904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874659468174352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6665562836918 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5416.933470997 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6714268459520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103406120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56873155 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197945171277824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56883298 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665903398192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6675636078080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6675977502208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23781048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚879806793673216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191230902818304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46532967 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6714165287424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874659485213712 _step‚471 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚886482771175424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103416265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚886482412711936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13747033 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320×LbÓ\2™- _wandbruntime‚5417  _runtime‚5417è~ߥÎ$šñM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191264083088384 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46540758 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚879955068175360 % _timestamp‚1.7845304532831552e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46540760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚886632155783680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚879955051136000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6715384357376 _runtime‚5417.551591201 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6677087608320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197979467445760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6715281192448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56882811 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚879955051136000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103423569 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10035471 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874806915881184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103433728 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚881473570066994 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6677087608320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46540758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13749561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56882811 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6666671225170 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103433728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23785035 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23784758 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46540760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13749287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197979345668608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191264064476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10035474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103423569 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10035471 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874806915881184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103433728 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚881473570066994 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6677087608320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46540758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13749561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56882811 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6666671225170 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103433728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23785035 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23784758 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46540760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13749287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197979345668608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191264064476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10035474ØÐûxƒ2œ- _wandbruntime‚5417  _runtime‚5417îâÕD „ 2loss/SRqqt1/assignment_loss‚0.5321918273959015 epoch‚14 % _timestamp‚1.7845304532847657e+09 /loss/FRt2/detection_loss‚0.5401206976489615 1loss/SRqqt2/detection_loss‚0.9578492136002539 _step‚473 3loss/FBt1/assignment_loss‚0.0010154182202605417 1loss/FBt2/assignment_loss‚0.08251983137704183 /loss/FBt1/detection_loss‚0.5623027141627972 /loss/FBt2/detection_loss‚0.9315772469382029 trainer/global_step‚5749 _runtime‚5417.552625662 0loss/FRt1/assignment_loss‚0.7089668709815033 2loss/SRqqt2/assignment_loss‚0.5627350863113908 /loss/FRt1/detection_loss‚1.3003952107299972 1loss/SRqqt1/detection_loss‚0.5995804121740047 %loss/total_loss‚8.502024864372427 0loss/FRt2/assignment_loss‚1.7227703348321108EÀµYž‰ /loss/FBt2/detection_loss‚0.9315772469382029 trainer/global_step‚5749 _runtime‚5417.552625662 0loss/FRt1/assignment_loss‚0.7089668709815033 2loss/SRqqt2/assignment_loss‚0.5627350863113908 /loss/FRt1/detection_loss‚1.3003952107299972 1loss/SRqqt1/detection_loss‚0.5995804121740047 %loss/total_loss‚8.502024864372427 0loss/FRt2/assignment_loss‚1.7227703348321108 2loss/SRqqt1/assignment_loss‚0.5321918273959015 epoch‚14 % _timestamp‚1.7845304532847657e+09 /loss/FRt2/detection_loss‚0.5401206976489615 1loss/SRqqt2/detection_loss‚0.9578492136002539 _step‚473 3loss/FBt1/assignment_loss‚0.0010154182202605417 1loss/FBt2/assignment_loss‚0.08251983137704183 /loss/FBt1/detection_loss‚0.5623027141627972Ùt3Ç號‚(:Ž ¡Œ÷Ò˜¹¯Þ! disk.nvme0c0n1.in‚ 10.06640625' disk.nvme0c0n1.out‚32937.3095703125% disk.nvme0n1.out‚32937.3095703125 disk.nvme1n1.in‚ 3.08203125 proc.memory.rssMB‚ 2925.5625 proc.cpu.threads‚17 network.sent‚ 9.24682e+06) disk./.usagePercent‚76.17972205776556% disk./.usageGB‚1169.5491828918457 disk.md0.in‚0 disk.dm-0.in‚ 13.1484375 disk.dm-1.in‚0 disk.rbd1.in‚0& disk.nvme1n1.out‚17555.67529296875 disk.dm-0.out‚42581.30859375" disk.dm-1.out‚7596.92431640625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0$ memory_percent‚97.16157913208008& proc.memory.availableMB‚ 465.046875 cpu‚25.69245846632906 disk.nvme0n1.in‚ 10.06640625 network.recv‚ 1.39604e+06 disk.md0.out‚ 146.91796875 disk.rbd0.in‚0( disk.nvme1c1n1.out‚17555.67529296875* proc.memory.percent‚17.8562164306640627û  ‚(:’ ¡Œ÷Òü÷ôÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.746' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.248666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U육‚(:’ °Œ÷Ò ç¢Þ% disk.nvme0n1.out‚33068.5498046875 disk.rbd0.out‚0 network.recv‚ 1.400158e+06) disk./.usagePercent‚76.17555862722442' disk.nvme0c0n1.out‚33068.5498046875 disk.nvme0n1.in‚ 10.06640625 disk.rbd1.out‚0 disk.nvme1n1.in‚ 3.08203125" disk.dm-1.out‚7596.99072265625( disk.nvme1c1n1.out‚17617.68994140625 disk.md0.out‚ 147.2421875$ memory_percent‚97.16172218322754' proc.memory.availableMB‚ 465.0234375 proc.memory.rssMB‚ 2925.5625* proc.memory.percent‚17.856216430664062 network.sent‚ 9.297111e+06$ disk./.usageGB‚1169.485263824463 disk.dm-0.in‚ 13.1484375" disk.dm-0.out‚42773.7724609375& disk.nvme1n1.out‚17617.68994140625 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.06640625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.in‚0 disk.md0.in‚0 cpu‚25.691885055132712 proc.cpu.threads‚17uÿò#“¢‚(:ˆ °Œ÷Ò°äòÞ gpu.0.gpu‚59.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚255.741' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.247 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼÷r€–£‚(:‹ ¿Œ÷ÒøŠ™Þ& proc.memory.availableMB‚ 464.703125 proc.memory.rssMB‚ 2925.5625* proc.memory.percent‚17.856216430664062 network.sent‚ 9.300075e+06& disk.nvme1c1n1.out‚17618.146484375 disk.rbd1.in‚0 disk.nvme1n1.in‚ 3.08203125$ disk.nvme1n1.out‚17618.146484375 disk.md0.out‚147.375 disk.nvme0n1.in‚ 10.06640625 disk.rbd0.out‚0 disk.md0.in‚0$ memory_percent‚97.16367721557617 cpu‚25.69126892117377 proc.cpu.threads‚17 network.recv‚ 1.403861e+06 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.out‚0# disk.dm-0.out‚42774.94482421875 disk.dm-1.in‚0! disk.dm-1.out‚7597.0751953125( disk.nvme0c0n1.out‚33069.67236328125 disk.rbd0.in‚0) disk./.usagePercent‚76.17555986959448% disk./.usageGB‚1169.4852828979492 disk.dm-0.in‚ 13.1484375! disk.nvme0c0n1.in‚ 10.06640625& disk.nvme0n1.out‚33069.67236328125vØõFž¤‚(:“ ¿Œ÷Òª¦íÞ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.672' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.89066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´øð†2¥- _wandbruntime‚5460  _runtime‚5460›øébɦà % _timestamp‚1.7845304961006603e+09 _runtime‚5460.369560554 _step‚474 ' lr-AdamW/pg1‚0.00014734637354989397 ' lr-AdamW/pg2‚0.00014734637354989397 trainer/global_step‚5799=ÇÓwÎ§È _step‚474 ' lr-AdamW/pg1‚0.00014734637354989397 ' lr-AdamW/pg2‚0.00014734637354989397 trainer/global_step‚5799 % _timestamp‚1.7845304961006603e+09 _runtime‚5460.369560554Ú6 ¼ø2¨- _wandbruntime‚5460  _runtime‚5460›[<ì'©æO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5460.369653254 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6731205179904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚887220501741056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6721386611600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882031018587312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887220501741056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784530496101328e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104280422 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6770086268416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6731205179904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192889897710080 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚5799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46922617 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57367950 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13861057 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6731546604032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10120437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6721046050326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57367950 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23981787 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23981494 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13861347 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882031001547952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6769980999168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚888752047598278 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887220518780416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192889916322304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚893951706920960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6731546604032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46922615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46922615 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199659878709248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184ò ¡ ˜ªëO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6721386611600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882031018587312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887220501741056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784530496101328e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104280422 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6770086268416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6731205179904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192889897710080 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚5799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46922617 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57367950 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13861057 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6731546604032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10120437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6721046050326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57367950 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23981787 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23981494 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13861347 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882031001547952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6769980999168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚888752047598278 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887220518780416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192889916322304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚893951706920960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6731546604032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46922615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46922615 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199659878709248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5460.369653254 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6731205179904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚887220501741056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248ÛB{%²2«- _wandbruntime‚5460  _runtime‚5460;˜Ã½÷&¬ñM trainer/global_step‚5799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6732315278848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23985565 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _runtime‚5460.992998907 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57377620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57377620 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46930410 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894101449992704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46930408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚199694298677760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚887368793282560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845304967246528e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚476 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894101449992704 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887368793282560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104308030 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚888900944254748 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10122171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882178449254784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6732656710144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6771202085376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6722494999964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104297871 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192923077980160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚888900586654002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6771096823296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887368776243200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46930410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104308030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199694174803456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104297871 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57367463 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _runtime‚5460.992998907 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57377620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57377620 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46930410 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894101449992704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46930408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚199694298677760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚887368793282560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845304967246528e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚476 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894101449992704 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887368793282560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104308030 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚888900944254748 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10122171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882178449254784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6732656710144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6771202085376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6722494999964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104297871 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192923077980160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚888900586654002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6771096823296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887368776243200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46930410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104308030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199694174803456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104297871 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57367463 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23985565 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ÜÞâŽm2®- _wandbruntime‚5460  _runtime‚5460£@ªÖ ¯… 1loss/SRqqt1/detection_loss‚0.5805956008164173 % _timestamp‚1.7845304967257435e+09 _runtime‚5460.993535348 2loss/FBt1/assignment_loss‚0.006959914100144291 1loss/SRqqt2/detection_loss‚0.9585974798723437 0loss/FBt1/detection_loss‚0.44136903218345763 trainer/global_step‚5799 0loss/FRt1/assignment_loss‚0.6711357209048046 0loss/FRt2/assignment_loss‚1.6356897800708805 1loss/SRqqt1/assignment_loss‚0.489839776682556 2loss/SRqqt2/assignment_loss‚0.4885968202773398 3loss/FBt2/assignment_loss‚0.0015739867633997324 /loss/FRt1/detection_loss‚1.4174862769452352 epoch‚14 /loss/FRt2/detection_loss‚0.5419767503156393 /loss/FBt2/detection_loss‚0.5085289223740498 %loss/total_loss‚7.742350061306268 _step‚477‘z¬°Š %loss/total_loss‚7.742350061306268 _step‚477 1loss/SRqqt1/detection_loss‚0.5805956008164173 % _timestamp‚1.7845304967257435e+09 _runtime‚5460.993535348 2loss/FBt1/assignment_loss‚0.006959914100144291 1loss/SRqqt2/detection_loss‚0.9585974798723437 0loss/FBt1/detection_loss‚0.44136903218345763 trainer/global_step‚5799 0loss/FRt1/assignment_loss‚0.6711357209048046 0loss/FRt2/assignment_loss‚1.6356897800708805 1loss/SRqqt1/assignment_loss‚0.489839776682556 2loss/SRqqt2/assignment_loss‚0.4885968202773398 3loss/FBt2/assignment_loss‚0.0015739867633997324 /loss/FRt1/detection_loss‚1.4174862769452352 epoch‚14 /loss/FRt2/detection_loss‚0.5419767503156393 /loss/FBt2/detection_loss‚0.5085289223740498Ý\“±‚(:ˆ ÎŒ÷Ò¸ÞæÝ disk.nvme1c1n1.in‚ 3.08203125 disk.md0.out‚ 147.85546875" disk.dm-0.out‚42784.5556640625$ memory_percent‚97.16386795043945 proc.cpu.threads‚17# disk.nvme1n1.out‚17620.69921875 network.sent‚ 9.305045e+06) disk./.usagePercent‚76.17556831771087% disk./.usageGB‚1169.4854125976562 disk.rbd0.out‚0 disk.nvme0n1.in‚ 10.06640625 proc.memory.rssMB‚ 2925.5625 cpu‚25.691023478634133 network.recv‚ 1.406889e+06 disk.nvme1n1.in‚ 3.08203125 disk.md0.in‚0 disk.dm-0.in‚ 13.1484375 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 10.06640625% disk.nvme0c0n1.out‚33077.83203125# disk.nvme0n1.out‚33077.83203125 disk.rbd0.in‚0% disk.nvme1c1n1.out‚17620.69921875 disk.rbd1.in‚0 disk.rbd1.out‚0! disk.dm-1.out‚7597.1494140625& proc.memory.availableMB‚ 464.671875* proc.memory.percent‚17.856216430664062?¥ž²‚(:“ ÎŒ÷Ò£ÛÞ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚287.177' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.72566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÕX×”³‚(:‰ ÝŒ÷Ò¼‚’Þ disk.rbd1.out‚0 disk.md0.in‚0 disk.dm-0.in‚ 13.1484375# disk.dm-0.out‚42785.49365234375$ disk./.usageGB‚1169.485507965088! disk.nvme0c0n1.in‚ 10.06640625$ disk.nvme0c0n1.out‚ 33079.1015625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme1n1.in‚ 3.08203125 disk.dm-1.in‚0$ memory_percent‚97.16386795043945 disk.nvme0n1.in‚ 10.06640625" disk.nvme0n1.out‚ 33079.1015625 disk.nvme1c1n1.in‚ 3.08203125& disk.nvme1n1.out‚17621.36865234375 proc.memory.rssMB‚ 2925.5625* proc.memory.percent‚17.856216430664062 proc.cpu.threads‚17) disk./.usagePercent‚76.17557452956117( disk.nvme1c1n1.out‚17621.36865234375 disk.md0.out‚ 148.28125! disk.dm-1.out‚7597.2451171875& proc.memory.availableMB‚ 464.671875 cpu‚25.691135421622338 network.sent‚ 9.368662e+06 network.recv‚ 1.411181e+06 disk.rbd0.in‚0"wWž´‚(:“ ÝŒ÷ÒàÚêÞ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.955' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.65166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03¬3c–µ‚(:‹ ìŒ÷ÒŽ‹Þ network.recv‚ 1.414368e+06 disk.rbd0.in‚0& disk.nvme1c1n1.out‚17622.033203125 disk.dm-0.in‚ 13.1484375! disk.nvme0c0n1.in‚ 10.06640625 proc.memory.rssMB‚ 2925.5625 cpu‚25.69134191144447% disk./.usageGB‚1169.4855270385742 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.in‚0 disk.md0.in‚0! disk.dm-0.out‚42787.455078125! disk.dm-1.out‚7597.3154296875& proc.memory.availableMB‚ 464.671875) disk./.usagePercent‚76.17557577193122 disk.nvme0n1.in‚ 10.06640625$ disk.nvme1n1.out‚17622.033203125 disk.md0.out‚ 148.57421875 disk.dm-1.in‚0* proc.memory.percent‚17.856216430664062 network.sent‚ 9.37438e+06% disk.nvme0n1.out‚33081.1103515625 disk.rbd1.out‚0 disk.nvme1n1.in‚ 3.08203125' disk.nvme0c0n1.out‚33081.1103515625$ memory_percent‚97.16386795043945 proc.cpu.threads‚17ßí·Fž¶‚(:“ ìŒ÷ÒšÌÌÞ gpu.0.gpu‚66.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚124.877' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.62566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùèC2·- _wandbruntime‚5506  _runtime‚5506’o¿ ɸà _runtime‚5506.151786684 _step‚478 ' lr-AdamW/pg1‚0.00014679909508247927 ' lr-AdamW/pg2‚0.00014679909508247927 trainer/global_step‚5849 % _timestamp‚1.7845305418835223e+09šÑYüÎ¹È _runtime‚5506.151786684 _step‚478 ' lr-AdamW/pg1‚0.00014679909508247927 ' lr-AdamW/pg2‚0.00014679909508247927 trainer/global_step‚5849 % _timestamp‚1.7845305418835223e+09Þ{~2º- _wandbruntime‚5506  _runtime‚5506Rlþ%»æO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201374672758272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194548929826304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚896179421785920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24182737 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6825761544192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚ 5506.15528459 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚479 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚894634226848256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24183028 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201374805028352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6776869825008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57852602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.òÝïÇon_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚901421001129984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57852602 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚894634226848256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57842459 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6787115705856 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336¹tåñ'¼ëO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6786774281728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194548911214080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13975888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47312267 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13975600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚889402551960912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚901421359593472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105154724 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6825875202048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7845305418872426e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚894634243887616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6787115705856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57842459 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47312265 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57842459 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6787115705856 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201374672758272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194548929826304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚896179421785920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24182737 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6825761544192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚ 5506.15528459 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚479 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚894634226848256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24183028 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201374805028352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6776869825008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57852602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚901421001129984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57852602 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚894634226848256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚901570385731072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚894782501350400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13978187 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6787884380672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47320058 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6787884380672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6788225811968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13977894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24187061 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194582091484160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚894782518389760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚896327960841756 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47320058 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10208871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57852115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24186765 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6826986546176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚480 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182332 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105172173 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194582110096384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6777637651986 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57862272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47320060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889549982628384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚894782501350400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6826881284096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0U8Ôêü¿öM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47320060 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57862272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚901570744201728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13978187 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6787884380672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47320058 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6787884380672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6788225811968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13977894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24187061 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194582091484160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚894782518389760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚896327960841756 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47320058 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10208871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57852115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24186765 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6826986546176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚480 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182332 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105172173 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 c¤W¤ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194582110096384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6777637651986 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57862272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47320060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889549982628384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚894782501350400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6826881284096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201409096642560 trainer/global_step‚5849 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10208874 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚901570385731072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚894782501350400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6842343383552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58327111 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚5551.214299988 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚908890653802496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6881706859520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47701917 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47701915 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14089324 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6881603687424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚896774085334512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚896774068295152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6832353038416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14089612 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908890295339008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6842343383552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908890653802496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6842684807680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584Í#ÿò'ÎìO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚896774085334512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚896774068295152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6832353038416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14089612 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908890295339008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6842343383552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908890653802496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6842684807680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58337254 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24383161 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6842343383552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58327111 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚5551.214299988 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚908890653802496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6881706859520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47701917 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47701915 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14089324 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6881603687424 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24387134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14091840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6843453482496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58346924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10295574 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚896921498962624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106056634 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚902196243496960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24387414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203123824153600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6882824427520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚903754977428764 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845305875774953e+09 _runtime‚5551.845796369 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6843453482496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106046475 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47709710 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47709710 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6833461426780 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58336767 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6843794913792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909039679940096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚896921516001984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106056634 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47709708 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902196226457600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6833120865394 _step‚484 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902196243496960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203123948027904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58346924 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902196226457600Ú| àü&ÑöM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902196226457600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6833120865394 _step‚484 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902196243496960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203123948027904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58346924 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902196226457600 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106046475 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909040038410752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14091563 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24387134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14091840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6843453482496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58346924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10295574 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚896921498962624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106056634 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚902196243496960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24387414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203123824153600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6882824427520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚903754977428764 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845305875774953e+09 _runtime‚5551.845796369 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6843453482496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106046475 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47709710 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47709710 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6833461426780 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58336767 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6843794913792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909039679940096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚896921516001984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106056634 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47709708 B¦äžØ‚(:“ ·÷Ò¬†§Þ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.949' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.64966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K6„kŸÙ‚(:” Æ÷Ò›Î°Þ disk.rbd0.out‚0% disk.nvme1n1.out‚17625.1962890625 disk.md0.in‚0 disk.dm-0.in‚ 13.1484375 disk.dm-1.in‚0% disk./.usageGB‚1169.4857597351074 disk.nvme1n1.in‚ 3.08203125# disk.dm-0.out‚42794.55224609375( disk.nvme0c0n1.out‚33088.88623046875& disk.nvme0n1.out‚33088.88623046875$ memory_percent‚97.15516567230225( proc.memory.availableMB‚ 466.09765625 proc.memory.rssMB‚ 2925.5625 disk.nvme1c1n1.in‚ 3.08203125 disk.md0.out‚ 150.07421875! disk.nvme0c0n1.in‚ 10.06640625* proc.memory.percent‚17.856216430664062) disk./.usagePercent‚76.17559092884595' disk.nvme1c1n1.out‚17625.1962890625 disk.rbd1.in‚0 disk.rbd1.out‚0! disk.dm-1.out‚7597.8310546875 disk.nvme0n1.in‚ 10.06640625 cpu‚25.694462264201913 proc.cpu.threads‚17 network.sent‚ 9.51789e+06 network.recv‚ 1.435593e+06 disk.rbd0.in‚0@È#CžÚ‚(:“ Æ÷Ò¨Û‰ß gpu.0.gpu‚95.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚255.163' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.05433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÛI2Û- _wandbruntime‚5595  _runtime‚5595~dÉÜà ' lr-AdamW/pg2‚0.00014569351173680553 trainer/global_step‚5949 % _timestamp‚1.7845306314631925e+09 _runtime‚5595.731444964 _step‚486 ' lr-AdamW/pg1‚0.00014569351173680553Yø&JÎÝÈ ' lr-AdamW/pg1‚0.00014569351173680553 ' lr-AdamW/pg2‚0.00014569351173680553 trainer/global_step‚5949 % _timestamp‚1.7845306314631925e+09 _runtime‚5595.731444964 _step‚486æ©BÔ–2Þ- _wandbruntime‚5595  _runtime‚5595tø69ßçO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6887495690550 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚909461694102016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5595.732196905 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916359589548032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937469465600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚904145618708112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚487 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚909461677062656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10380540 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6897912485376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14203420 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58811763 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚909461677062656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197866956834304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58811763 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚909461694102016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6887836251824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚5949 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48091567 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6898253909504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916359948011520 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚911033097359302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916359948011520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937574734848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106903328 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10380537 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106913473 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48091567 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚911033454959936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897912485376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204804531569152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48091565 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24583957 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197866938222080 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6898253909504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916359589548032Ùtÿ¦ò'àìO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚911033097359302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916359948011520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937574734848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106903328 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10380537 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106913473 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48091567 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚911033454959936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897912485376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204804531569152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48091565 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24583957 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197866938222080 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6898253909504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916359589548032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6887495690550 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚909461694102016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5595.732196905 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916359589548032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937469465600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚904145618708112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚487 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚909461677062656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10380540 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6897912485376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14203420 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58811763 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚909461677062656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197866956834304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58811763 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚909461694102016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6887836251824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚5949 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48091567 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6898253909504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916359948011520 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0çúHi2á- _wandbruntime‚5596  _runtime‚5596å‚>÷&âñM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚911181636415026 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚909609951564800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106930936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6888944640188 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48099360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48099358 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6888604078802 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58831576 % _timestamp‚1.7845306320915756e+09 _runtime‚5596.359893368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚909609968604160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106930936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6899364015616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106920777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14205967 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24587962 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204838704064512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10382274 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58821419 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14205691 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904293032336224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48099360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58831576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48099358 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916509332619776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚911181994015772 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚916509332619776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916508974149120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938688737280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6899364015616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6899022584320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938585572352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106920777 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚909609951564800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204838825841664i’}”$ãöM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚5949 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106930936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6888944640188 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48099360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48099358 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6888604078802 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58831576 % _timestamp‚1.7845306320915756e+09 _runtime‚5596.359893368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚909609968604160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106930936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6899364015616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106920777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14205967 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24587962 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204838704064512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10382274 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58821419 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14205691 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904293032336224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48099360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58831576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48099358 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916509332619776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚911181994015772 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚916509332619776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916508974149120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938688737280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6899364015616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6899022584320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938585572352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106920777 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚909609951564800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204838825841664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚911181636415026 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚909609951564800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚923828883757056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107777630 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚923828883757056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10467237 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚911517135042352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚491 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6942978903958 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6943319465232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107787775 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918460471546944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916875402169856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206519366362112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48481217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206519246674944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24785349 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59306558 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚923829242220544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916875419209216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6993294948864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5999 _runtime‚5640.816501467 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚911517152081712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48481215 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48481215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14317827 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚916875419209216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚918460113946310 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚923829242220544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6953823011328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452’£T¤ò'òìO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107777630 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚923828883757056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10467237 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚911517135042352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚491 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6942978903958 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6943319465232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107787775 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918460471546944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916875402169856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206519366362112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48481217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206519246674944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24785349 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59306558 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚923829242220544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916875419209216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6993294948864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5999 _runtime‚5640.816501467 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚911517152081712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48481215 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48481215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14317827 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚916875419209216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚918460113946310 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚923829242220544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6953823011328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14318109 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10467240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59306558 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚923828883757056ë×ÄWm2ó- _wandbruntime‚5641  _runtime‚5641â±ú¬ôðM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199559150608384 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚923978268358144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59306071 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14320123 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6944087292210 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚923978626828800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6954591686144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6994518246912 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6954933117440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6954933117440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59316228 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚911664582749184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚911664565709824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48489008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6994404596224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚5999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10468971 _step‚492 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107795079 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚5641.454344313 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6954591686144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199559131996160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24789094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚917023693711360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚923978626828800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6944427853596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚917023676672000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107805238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14320418 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚917023676672000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚917023693711360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚918609010602780 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184däÏ­û&õõM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6944087292210 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚923978626828800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6954591686144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6994518246912 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6954933117440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6954933117440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59316228 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚911664582749184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚911664565709824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48489008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6994404596224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚5999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10468971 _step‚492 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107795079 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚5641.454344313 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6954591686144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199559131996160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24789094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚917023693711360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚923978626828800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6944427853596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚917023676672000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107805238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14320418 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚917023676672000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚917023693711360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚918609010602780 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199559150608384 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚923978268358144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59306071 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14320123ìK]ü²2ö- _wandbruntime‚5641  _runtime‚5641Q' ÷„ /loss/FRt1/detection_loss‚1.4631736114363243 /loss/FBt2/detection_loss‚0.9853141608719642 %loss/total_loss‚8.830144812692893 trainer/global_step‚5999 % _timestamp‚1.7845306771880069e+09 0loss/FRt1/assignment_loss‚0.7600212983530144 1loss/FBt1/assignment_loss‚0.03009830341920967 1loss/SRqqt1/detection_loss‚0.6327479754173972 epoch‚14 _runtime‚5641.455975534 _step‚493 2loss/SRqqt1/assignment_loss‚0.5433229472806569 2loss/SRqqt2/assignment_loss‚0.5589934222290921 3loss/FBt2/assignment_loss‚0.0018273991525898754 1loss/SRqqt2/detection_loss‚0.8796996983269659 /loss/FBt1/detection_loss‚0.6427638319055404 0loss/FRt2/assignment_loss‚1.7945118784665104 /loss/FRt2/detection_loss‚0.5376702858336274Óâ0Ñø‰ 1loss/SRqqt2/detection_loss‚0.8796996983269659 /loss/FBt1/detection_loss‚0.6427638319055404 0loss/FRt2/assignment_loss‚1.7945118784665104 /loss/FRt2/detection_loss‚0.5376702858336274 /loss/FRt1/detection_loss‚1.4631736114363243 /loss/FBt2/detection_loss‚0.9853141608719642 %loss/total_loss‚8.830144812692893 trainer/global_step‚5999 % _timestamp‚1.7845306771880069e+09 0loss/FRt1/assignment_loss‚0.7600212983530144 1loss/FBt1/assignment_loss‚0.03009830341920967 1loss/SRqqt1/detection_loss‚0.6327479754173972 epoch‚14 _runtime‚5641.455975534 _step‚493 2loss/SRqqt1/assignment_loss‚0.5433229472806569 2loss/SRqqt2/assignment_loss‚0.5589934222290921 3loss/FBt2/assignment_loss‚0.0018273991525898754íØöaçžù‚(:“ ‚Ž÷Ò„ÒöÝ disk.nvme1n1.in‚ 3.08203125 disk.dm-1.in‚0' proc.memory.availableMB‚ 448.2265625( proc.memory.percent‚17.9610013961792 disk.rbd1.out‚0 network.sent‚ 9.593201e+06) disk./.usagePercent‚76.17558968647589( disk.nvme0c0n1.out‚33095.11474609375 disk.rbd0.out‚0% disk.nvme1n1.out‚17628.1669921875 disk.md0.out‚ 151.703125! disk.dm-1.out‚7598.2529296875$ disk./.usageGB‚1169.485740661621 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.md0.in‚0$ memory_percent‚97.26424217224121# proc.memory.rssMB‚ 2942.73046875 proc.cpu.threads‚17 network.recv‚ 1.450246e+06! disk.nvme0c0n1.in‚ 10.06640625 disk.nvme1c1n1.in‚ 3.08203125' disk.nvme1c1n1.out‚17628.1669921875 disk.dm-0.in‚ 13.1484375# disk.dm-0.out‚42799.84912109375 cpu‚25.702195415692813 disk.nvme0n1.in‚ 10.06640625& disk.nvme0n1.out‚33095.11474609375sÌ&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015159886330366135 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961622357368469 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11323310434818268 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.9012340307235718 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1499825268983841 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8514739274978638 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7398648858070374 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8422192931175232 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9450467228889465 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5709014534950256 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6404324769973755 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16921645402908325 /particle/accuracy_4_of_4‚0.9544776678085327 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7125040292739868 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4969068169593811 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 epoch‚14 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14672844111919403 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.058571942150592804 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1516486406326294 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5155195593833923 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1035604253411293 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00911637581884861 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953667521476746 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5866173505783081 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8642290234565735 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.332684189081192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029171645641326904 +jet/accuracy_3_of_4‚0.08124461024999619 jet/accuracy_3_of_5‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915412068367004 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13646464049816132 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.871353030204773 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995708167552948 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12916257977485657 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06743066757917404 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6026907563209534 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5460982322692871 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5279166102409363 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7219956517219543 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3541666865348816 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1309739500284195 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12797629833221436 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_1‚0.2714168131351471 *jet/accuracy_3_of_3‚0.0678911805152893 &particle/f_score‚0.630336582660675 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5570796132087708 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6898148059844971 trainer/global_step‚5999 _runtime‚5659.148254378 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964693784713745 *jet/accuracy_2_of_3‚0.5226977467536926 jet/accuracy_1_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8574074506759644 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03044835664331913 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19561660289764404 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6662041544914246 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14982950687408447 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8241995573043823 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021443771198391914 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253424406051636 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8589569330215454 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.01782079041004181 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13830827176570892 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5158273577690125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5930736064910889 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5709014534950256 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43919816613197327 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023542925715446472 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5269116759300232 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5204166769981384 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6264133453369141 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9967285990715027 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3887760639190674 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6556459069252014 +jet/accuracy_2_of_2‚0.13767549395561218 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7344502210617065 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.662188708782196 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7362110614776611 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19193610548973083 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.643390953540802 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004827677272260189 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20916961133480072 (loss/val_total_loss‚8.28726419658527 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25904929637908936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989761233329773 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5669425129890442 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.42643341422080994 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3347843885421753 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5577340722084045 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.164166659116745 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14596304297447205 *jet/accuracy_2_of_4‚0.7511869668960571 /particle/accuracy_3_of_4‚0.9968254566192627 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.88981693983078 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6898148059844971 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7845306948795753e+09 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.882555365562439 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13646464049816132 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.6936335563659668 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21880193054676056 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5511761903762817 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1372717320919037 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922282099723816 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.577556312084198 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5814101099967957 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.07956819981336594 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12377320230007172 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.43922218680381775 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )jet/accuracy_1_of_3‚0.880316436290741 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt1SRqqt1FBt/event_purity‚ºB¦ã0.497229665517807 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7806689739227295 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24354688823223114 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6271113753318787 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _step‚494ø’{Eª=ý¤{ 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11810463666915894 6validation_average_jet_accuracy‚0.3165481686592102 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3515625 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4467592239379883 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11379116028547287 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9897027015686035 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5812872648239136 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.08892212808132172 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02101125381886959 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6221404075622559 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9830824732780457 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5657669901847839 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6480286717414856 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5792281627655029 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7401385307312012 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.705742597579956 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_3‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11520251631736755 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015159886330366135 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961622357368469 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11323310434818268 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.9012340307235718 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1499825268983841 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8514739274978638 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7398648858070374 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8422192931175232 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9450467228889465 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5709014534950256 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6404324769973755 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16921645402908325 /particle/accuracy_4_of_4‚0.9544776678085327 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7125040292739868 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4969068169593811 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 epoch‚14 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14672844111919403 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.058571942150592804 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1516486406326294 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5155195593833923 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1035604253411293 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00911637581884861 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953667521476746 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5866173505783081 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8642290234565735 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.332684189081192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029171645641326904 +jet/accuracy_3_of_4‚0.08124461024999619 jet/accuracy_3_of_5‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915412068367004 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13646464049816132 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.871353030204773 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995708167552948 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12916257977485657 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06743066757917404 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6026907563209534 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5460982322692871 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5279166102409363 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7219956517219543 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3541666865348816 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1309739500284195 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12797629833221436 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_1‚0.2714168131351471 *jet/accuracy_3_of_3‚0.0678911805152893 &particle/f_score‚0.630336582660675 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5570796132087708 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6898148059844971 trainer/global_step‚5999 _runtime‚5659.148254378 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964693784713745 *jet/accuracy_2_of_3‚0.5226977467536926 jet/accuracy_1_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8574074506759644 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03044835664331913 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19561660289764404 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6662041544914246 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14982950687408447 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8241995573043823 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021443771198391914 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253424406051636 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8589569330215454 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.01782079041004181 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13830827176570892 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5158273577690125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5930736064910889 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5709014534950256 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43919816613197327 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023542925715446472 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5269116759300232 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5204166769981384 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6264133453369141 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9967285990715027 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3887760639190674 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6556459069252014 +jet/accuracy_2_of_2‚0.13767549395561218 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7344502210617065 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.662188708782196 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7362110614776611 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19193610548973083 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.643390953540802 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004827677272260189 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20916961133480072 (loss/val_total_loss‚8.28726419658527 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25904929637908936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989761233329773 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5669425129890442 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.42643341422080994 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3347843885421753 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5577340722084045 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.164166659116745 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14596304297447205 *jet/accuracy_2_of_4‚0.7511869668960571 /particle/accuracy_3_of_4‚0.9968254566192627 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.88981693983078 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6898148059844971 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7845306948795753e+09 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.882555365562439 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13646464049816132 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.6936335563659668 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21880193054676056 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5511761903762817 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1372717320919037 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922282099723816 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.577556312084198 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5814101099967957 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.07956819981336594 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12377320230007172 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.43922218680381775 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )jet/accuracy_1_of_3‚0.880316436290741 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.497229665517807 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7806689739227295 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24354688823223114 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6271113753318787 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _step‚494 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5990106463432312 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6566075086593628 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5755652189254761 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970045685768127 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948979616165161 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023542925715446472 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8241995573043823 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6528038382530212 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6123510599136353 )jet/accuracy_1_of_4‚0.965958833694458 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020289074629545212 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.847997784614563 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.019932758063077927îa †L þ‚(:• ‘Ž÷Ò–î‹Þ network.sent‚ 9.655529e+06) disk./.usagePercent‚76.17560111628043 disk.md0.out‚ 152.0546875# disk.dm-0.out‚42800.45068359375 disk.nvme0n1.in‚ 10.06640625& disk.nvme0n1.out‚33095.96435546875 disk.nvme1c1n1.in‚ 3.08203125$ memory_percent‚97.30157852172852 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚7598.4150390625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2948.01171875 cpu‚25.703462902845633 proc.cpu.threads‚17 disk.nvme1n1.in‚ 3.08203125 disk.rbd0.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 442.109375) proc.memory.percent‚17.99323558807373( disk.nvme0c0n1.out‚33095.96435546875 network.recv‚ 1.454624e+06% disk./.usageGB‚1169.4859161376953% disk.nvme1n1.out‚17628.8349609375 disk.dm-0.in‚ 13.1484375! disk.nvme0c0n1.in‚ 10.06640625' disk.nvme1c1n1.out‚17628.8349609375 disk.rbd1.out‚0ó“Â]žÿ‚(:“ ‘Ž÷ÒÅÐÞ gpu.0.gpu‚66.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.30733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Hžž€‚(:“  Ž÷Ò‚Þ% disk./.usageGB‚1169.4856758117676' disk.nvme1c1n1.out‚17629.8544921875% disk.nvme1n1.out‚17629.8544921875 disk.dm-1.in‚0! disk.dm-1.out‚7598.4619140625# proc.memory.rssMB‚ 2948.01171875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 10.06640625 disk.nvme1c1n1.in‚ 3.08203125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.md0.out‚ 152.28125) proc.memory.percent‚17.99323558807373 network.recv‚ 1.458418e+06& disk.nvme0n1.out‚33099.10400390625 disk.rbd0.out‚0 disk.nvme1n1.in‚ 3.08203125! disk.nvme0c0n1.in‚ 10.06640625$ memory_percent‚97.30007648468018( proc.memory.availableMB‚ 442.35546875 network.sent‚ 9.714454e+06 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 13.1484375# disk.dm-0.out‚42804.06298828125( disk.nvme0c0n1.out‚33099.10400390625 cpu‚25.70318936683593( disk./.usagePercent‚76.1755854624177Gµ¬ž‚(:“  Ž÷Ò™¿ùß gpu.0.gpu‚90.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.888' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.29599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@"šœ‚‚(:‘ ¯Ž÷Ò‰Ì÷Ý( proc.memory.availableMB‚ 442.33203125 network.recv‚ 1.462115e+06 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.nvme1n1.in‚ 3.08203125 disk.md0.out‚ 152.78515625! disk.nvme0c0n1.in‚ 10.06640625 proc.cpu.threads‚17$ memory_percent‚97.30021953582764# proc.memory.rssMB‚ 2948.01171875) proc.memory.percent‚17.99323558807373 cpu‚25.70427711659592 network.sent‚ 9.71762e+06) disk./.usagePercent‚76.17559863154031& disk.nvme0n1.out‚33100.43408203125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 3.08203125 disk.md0.in‚0 disk.dm-0.in‚ 13.1484375! disk.dm-0.out‚42804.974609375% disk./.usageGB‚1169.4858779907227 disk.nvme0n1.in‚ 10.06640625% disk.nvme1c1n1.out‚17630.61328125# disk.nvme1n1.out‚17630.61328125 disk.dm-1.in‚0" disk.dm-1.out‚7598.55224609375( disk.nvme0c0n1.out‚33100.43408203125zÛ: žƒ‚(:“ ¯Ž÷Ò¹†ÂÞ gpu.0.gpu‚71.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚257.557' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.85233333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qñ2„- _wandbruntime‚5703  _runtime‚5703Mˆ°ƒÇ…Á & lr-AdamW/pg1‚0.0001445734635971481 & lr-AdamW/pg2‚0.0001445734635971481 trainer/global_step‚6049 % _timestamp‚1.7845307393982666e+09 _runtime‚5703.667145222 _step‚495ß—zÔÌ†Æ & lr-AdamW/pg1‚0.0001445734635971481 & lr-AdamW/pg2‚0.0001445734635971481 trainer/global_step‚6049 % _timestamp‚1.7845307393982666e+09 _runtime‚5703.667145222 _step‚495ï%†B2‡- _wandbruntime‚5703  _runtime‚5703Ž=~_í'ˆçO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48993675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚5703.667469963 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108856234 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚926231471152640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚926231471152640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10558053 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845307393991637e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48993675 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7069980857856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚920809924424176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7019106527537 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7029368524800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚926231488192000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208277644437504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7070081932800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7029368524800 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25028068 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14470012 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933260839677440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933261198140928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7029709948928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚927829047991073 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25027781 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7018765966263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59862561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59872704 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108866379 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚6049 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10558056 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208277524750336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201207562504704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚927828690390439 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48993673 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201207543892480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚926231488192000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14469728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚920809941463536 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.a